Retrieving large batches of online game codes?

Question

In my current solution, a user triggers my web API from a web application of mine first, then my web API calls a 3rd party web service, and the results/codes are displayed on the web application. In the current design (web application > web API > 3rd party web service), the user must remain in front of the screen and continuously send requests to the API until he or she is able to successfully complete 20K codes. For example, the user must send requests to the API every 5 minutes for hours until he or she is able to get 20K codes in order to complete the task. In a single request, this third-party web API returns a maximum of 10 results/codes. (eg. quantity:1 => code:1, quantity:10 => codes:10) There are times it is needed to get 20.000 results/codes from this service.

The user has to stay in front of the screen for hours to get 20000 codes. This can be tedious work, requiring a great deal of dedication and focus. The aim of my project is to reduce the dependency on the user and reduce the total time spent with the service in the background. To achieve this, my project will automate the process of generating the codes, thus reducing the user's workload and freeing them up to focus on other tasks.

I implemented an IHostedService that calls this 3rd party web API in the background. I need your feedback in terms of speed, error handling and etc. How can I improve my solution? Do you have any comments?

Here is the Worker:

   using System.Data;
using System.Data.SqlClient;
using System.Diagnostics;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text;
using PurchaseRazerCodesMultipleRequestService.Modal;
using TrendyolGameCodeService.Modal;


namespace PurchaseRazerCodesMultipleRequestService
{
    public class Worker : BackgroundService
    {
        private readonly ILogger<Worker> _logger;
        private readonly IHttpClientFactory _httpClientFactory;
        private readonly IConfiguration _configuration;

        public Worker(ILogger<Worker> logger, IHttpClientFactory httpClientFactory, IConfiguration configuration)
        {
            _logger = logger;
            _httpClientFactory = httpClientFactory;
            _configuration = configuration;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            
            ///Random Timer
            var timer = new PeriodicTimer(TimeSpan.FromMinutes(new Random().Next(1, 2)));
            _logger.LogInformation("Timer: {timer}", DateTimeOffset.Now);
            while (await timer.WaitForNextTickAsync(stoppingToken))
            {
                await GetData();
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
            }
        }

        private async Task MakeRequestsToRemoteService(string productCode, long id, int amount, int bulkId)
        {
            if (id <= 0) throw new ArgumentOutOfRangeException(nameof(id));
            try
            {
                var httpClient = _httpClientFactory.CreateClient("RazerClient");

                //Token

                httpClient.DefaultRequestHeaders.Accept.Add(
                    new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded"));

                var tokenContent = new FormUrlEncodedContent(new[]
                {
                    new KeyValuePair<string, string>("username", "Test"),
                    new KeyValuePair<string, string>("password", "+REyN-#V5!_DgUn+y%hVj7VmyhN^+?%y+Qxkc-bLZR6$uqsYV")
                });

                using var tokenResponse = await httpClient.PostAsync(_configuration["Token:Production"], tokenContent);

                if ((int)tokenResponse.StatusCode == 200)
                {
                    var token = await tokenResponse.Content.ReadAsStringAsync();

                    //Call Razer Multi Requests

                    httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer",
                        token);
                    httpClient.DefaultRequestHeaders.Accept.Add(
                        new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded"));

                    #region Calculate quantity

                    var num = amount;
                    var firstNum = 50;
                    var secondNum = 50;

                    if (num < 100)
                    {
                        firstNum = (num + 1) / 2;
                        secondNum = num - firstNum;
                    }

                    #endregion

                    var quantities = new List<int> { firstNum, secondNum};
                    var cts = new CancellationTokenSource();
                    ParallelOptions parallelOptions = new()
                    {
                        MaxDegreeOfParallelism = 2,
                        CancellationToken = cts.Token
                    };
                    try
                    {
                        await Parallel.ForEachAsync(quantities, parallelOptions, async (quantity, ct) =>
                        {
                            var content = new FormUrlEncodedContent(new[]
                            {
                                new KeyValuePair<string, string>("productCode", productCode),
                                new KeyValuePair<string, string>("quantity", quantity.ToString()),
                                new KeyValuePair<string, string>("clientTrxRef", bulkId.ToString())
                            });

                            using var response =
                                await httpClient.PostAsync(_configuration["Razer:Production"], content, ct);

                            if ((int)response.StatusCode == 200)
                            {
                                var coupon = await response.Content.ReadFromJsonAsync<Root>(cancellationToken: ct);

                                _logger.LogInformation("REFERENCE ID: {referenceId}", coupon.ReferenceId);

                                await UpdateData(id);
                            }
                            else
                            {
                                _logger.LogError("Purchase ServiceError: {statusCode}",
                                    (int)response.StatusCode);
                            }
                        });
                    }
                    catch (OperationCanceledException ex)
                    {
                        _logger.LogError("Operation canceled: {Message}",
                            ex.Message);
                    }
                }
                else
                {
                    _logger.LogError("Token ServiceError: {statusCode}",
                        (int)tokenResponse.StatusCode);
                }
            }
            catch (Exception e)
            {
                _logger.LogError("Error: {Error} ", e.Message);
                
            }
        }

        private async Task GetData()
        {
            var sw = Stopwatch.StartNew();
            var connString = _configuration["ConnectionStrings:Default"];
            await using var sqlConnection = new SqlConnection(connString);
            sqlConnection.Open();

            await using var command = new SqlCommand { Connection = sqlConnection };
            const string sql = @"Select TOP 1 Id, BulkPurchaseRequestId, Amount, ProductCode from BulkPurchases where status = 0 ORDER BY NEWID()";
            command.CommandText = sql;

            
            try
            {
                await using var reader = await command.ExecuteReaderAsync();
                while (reader.Read())
                {
                    _logger.LogInformation(
                        "Order {Id}, {BulkId}, {Amount}, {ProductCode}",
                        reader.GetInt32(0), reader.GetInt32(1), reader.GetInt32(2), reader.GetString(3));

                    await MakeRequestsToRemoteService(reader.GetString(3).Trim(), reader.GetInt32(0), reader.GetInt32(2),reader.GetInt32(1));
                }
            }
            catch (SqlException exception)
            {
                _logger.LogError("Error: {Error} ", exception.Message);
                
            }

            sw.Stop();

            _logger.LogInformation($"******** ELAPSED TIME: {sw.Elapsed.TotalSeconds} seconds ********");
        }

        private async Task UpdateData(long id)
        {
            var connString = _configuration["ConnectionStrings:Default"];
            await using var sqlConnection = new SqlConnection(connString);
            sqlConnection.Open();

            await using var command = new SqlCommand { Connection = sqlConnection };
            const string sql = @"Update BulkPurchases set status = 1 where Id = @id";
            command.CommandText = sql;

            command.Parameters.Add(new SqlParameter("id", id));

            try
            {
                await command.ExecuteNonQueryAsync();
            }
            catch (SqlException exception)
            {
                _logger.LogError("Error: {Error} ", exception.Message);
                
            }
        }
    }
}

Edit

I have written a ASP.NET Core Web API that retrieves game codes from another external API. The process works like this: the game code and the number of codes to be retrieved are sent in the request. However, there is a restriction in the external API I am retrieving the game codes from, which is that only 10 game codes can be retrieved in one request. This process is currently being done in various chain stores' cash registers. Only one game code purchase transaction is made from the cash register.

However, there can be customers who want to bulk retrieve thousands of game codes online. To achieve this, I added a new method to the API to enable multiple purchases by looping. The requests are sent to the external API in small pieces, with 10 being the requested amount of game codes. This process works without any problems because each successful request and response is recorded in the database. This process is carried out through the ASP.NET Core interface and has a limitation: if the user inputs the amount of game codes requested through the interface, it takes a long time to retrieve thousands of game codes as the maximum is 100 (to avoid time-out issues, etc.).

To improve this situation, I created a worker service that operates in the background. The user inputs the total request through the web interface, which is converted into 100s and recorded in the database. The worker service retrieves these requests one by one randomly, then sends the requests to the API I created and then to the external API. The new process in the worker service is as follows: when 100 game code requests are made, the maximum parallelism is 2 and they are sent in Parallel.ForEachAsync, divided into 50/50. The requests are processed in the manner described in 10s, as previously mentioned. My concern here is if 100 game codes are successfully sent and retrieved, I update the related record in the database. However, if an error occurs somewhere in the process of processing the external API in 10s, my API will return a 500 error. I'm not exactly sure whether the Parallel.ForEachAsync will continue processing the other requests or if the operation will be cancelled. I was unable to test this scenario. What logic would be appropriate to construct here? Especially for the update scenario. Is there a way to mock the service in order to get errors once in a while so that I can test the logic?

Online game codes are used for purchasing items in online games

Answer 1

Orchestration vs Choreography

• Whenever you want to execute a sequence of actions then you can pipe them through:
  • choreography: Each and every action knows who is the next in the calling chain
    • Or in other words, they know how to interact with each other
  • orchestration: There is a an orchestrator which chains the actions after each other
    • Or in other words, they don't know how to interact with each other
• You have chosen the choreography which made your program harder to understand IMHO
  • None of your methods return anything rather they are calling each other
  • As a reviewer/maintainer it is harder to create a mental model
• If you would compose your actions as an orchestration
  • then it would be much easier to follow the data and execution flows
  • Here is a (pseudo) example

async Task ExecuteCoreAsync(CancellationToken token)
{
    var purchaseRequests = await GetPurchaseRequests(token);
    var accessToken = await RetrieveAccessToken(token);
    var purchaseIds = await ProcessPurchaseRequests(accessToken, purchaseRequests, token);
    await UpdateStatusOfPurchaseRequests(purchaseIds, token);
}

• The above code sequentially executes each actions
  • But from this mental model it is much easier to create a pipeline
    • with TPL Dataflow or System.Threading.Channels for example

Problem domain decomposition

• We can see that you have tried to split up the problem domain into smaller chunks
  • My problem here is that they are not on the same level of granularity
    • Compare MakeRequestsToRemoteService and UpdateData
• MakeRequestsToRemoteService has multiple responsibilities
  • Retrieving access token
  • Issuing requests against downstream system
  • Calling update on database records
• If you would try to aim for smaller composable chunks
  • then it is much easier to create actions with single responsibility
• BTW: based on the shared code fragment it seems like it is unnecessary to retrieve access token whenever you start to process a purchase request
  • IMHO it would be enough to retrieve only once

Distributed transaction

• According to my understanding your code is the glue between two subsystems (a database and a service) >> distributed
• I assume that you want to guarantee that either you can update records in both systems or in none of them >> transaction
  • Let's suppose that for some reason your last action the UpdateData fails for a given purchase
  • That would mean the database is not updated but the other subsystem is
  • Because of the purchase domain I assume that this inconsistency can cause problems
• You could for example do a compensation action against the service to undo/rollback the change to heal the consistency across multiple subsystems
• I would suggest read about SAGA pattern or Two/Three Phase Commit protocols