Large Scale Data Engineering Assignment Sample

Task 1: Voice2Text Application Architecture Design on AWS Cloud

Overview of Application Requirements

The Voice2Text application is designed to target users with a subscription in Europe, ensuring they can securely upload audio files to be transcribed into text format and sent to them via email. The architecture should prioritise low latency, high availability, security, and cost-effectiveness. The architecture also guarantees the privacy of storing the audio files, quick processing of transcription, and delivery of final results to the inbox of the users (Yashina et al., 2022).

For better academic guidance on technical topics, native assignment help provides a trusted assignment writing service tailored to complex coursework.

AWS Services and Functional Integration

The storage of audio files uploaded by the users is stored in Amazon S3 as secure storage. Buckets are set with an access limitation, which protects the audio data of each user as an individual user cannot access the audio data belonging to another user. The S3 bucket is set in such a way that on uploading an object an AWS Lambda function is triggered. The Lambda functions to start the process of transcription and put the metadata of the job into Amazon SQS to be processed asynchronously.

User authentication and authorization are carried out on Amazon Cognito. It limits the use of applications to registered users in the European areas and generates secure tokens that can provide a momentary way to upload the information to S3 (Kumar et al., 2021).

AWS Lambda reads the details of the jobs placed on SQS and calls Amazon Transcribe to transcribe them. Transcribe has been set up in the EU-based area which ensures that the product follows GDPR and local data management regulations. When the transcription is done, the results are again saved into Amazon S3, in the same secret bucket, but in a new structure in which only the system has rights to access.

Amazon RDS is the relational database where user metadata as well as transcription job statuses are stored. It provides support to multi-availability zone deployment in order to increase availability and resilience. In the meantime, sending results over email is done through Amazon SES. The text and its storage method are Top Hat, the text always transmits and the stored transcript is either an encrypted link to the stored transcript or a formatted email. Access between services is handled by IAM roles following least privilege principles and KMS takes care of encryption of data at rest.

User Workflow and Network Design

Figure 1: Voice2Text Architecture

When the registered user authenticates through the application, his session is authenticated by Amazon Cognito. When the front end delivers an audio data file to Amazon S3 using CloudFront, the file is saved in the Amazon S3. This is done by calling a Lambda which will validate the upload and insert metadata related to the jobs into the RDS, and also send a notification into the SQS queue. Another Lambda should be reading the queue, creating a Transcribe job, and when it finishes, another Lambda should be turning the transcript into a more human-friendly form and sending it to the user through SES (Chauhan and Arora, 2023). The diagram that comes along with this explanation shows how the AWS services can interact with the secure as well as efficient workflow.

Task 2: Scaling the WordPress Application

1. Introduction

This task is meant to change a default single-instance WordPress installation into a cloud-native structure that is capable of scaling, resilient, and secure cloud-native architecture. In the first instance, WordPress is run on an independent EC2 instance, which has a number of limitations such as low fault tolerance, limited scalability under average user load, and huge latency under peak traffic. Such deficiencies are not only a performance killer but also a reliability hazard to any application that has an uptime and responsiveness requirement. The goal of the exercise is to recognize such performance bottlenecks, implement auto-scaling strategies and make the service highly available by utilizing Load Balancers and multi-instance deployment. Also, the assignment aims to improve the security of the architecture by means of close IAM role configuration and network controls. Economical utilization is also an important factor that demands dynamically adjusted infrastructure to prevent the wasteful use of resources. This report explains the procedure, results, and measures that were taken to achieve those goals and create a healthy WordPress deployment on AWS.

2. Task A: Installing the Application

2.1 Installation Process

Figure 2: Successfully launch instance

WordPress program was introduced on AWS under Bitnami AMI which is pre-configured. In AWS Academy Learner Lab, Bitnami WordPress AMI (ami-0135570cf67844e2b) was used to trigger the EC2 deployment in the AWS environment. The t2.micro type is selected as the inexpensive optimizer when the test had to be conducted at the very beginning (Balayn et al. 2021). Creating and downloading key pair in format (pem) and named as wordpress-key pair so as to be able to connect using SSH. After the successful launch of the instance, the instance state appeared to be running, as well as, both status checks passed successfully. The WordPress landing page was opened in a browser using the public DNS which was confirmation that the web server was online. This confirmed that indeed the Bitnami stack had been successfully set up and also the instance was entirely functional and could be administered and also tested.

Figure 3: Inbound security

The security group to be used was called wordpress-sg and two inbound rules were created; HTTP (port 80) and SSH (port 22), both temporarily to allow traffic in all the sources to facilitate the creation of the system (Serra et al., 2023). After launching the instance user requested its application in the browser through the provided public DNS address. The confirmation of the successful launch of the instance was in the form of its status of a running instance and either of the system status checks succeeding. A browser was used to visit the public DNS, and the default Bitnami page of WordPress was seen, which confirmed the presumption that the Apache server and the WordPress stack were installed properly and were visible..

2.2 Access and Setup

Figure 4: Copying Public IPv4 DNS

In order to access the WordPress administrator panel, the extension /wp-admin was added at the end of the public DNS of the instance. Upon the completion of EC2 instance, Public IPv4 DNS was copied under the instance details screen of the AWS console. This is DNS address which will be a destination in accessing the WordPress site which user will deploy through a web browser. It is also necessary to verify the installation as well as access the /wp-admin dashboard.

Figure 5: Finding password

The default username (user) was applied and the password was accessed by querying the EC2 system log through the search term (Bitnami application password). This enabled users to log in to the dashboard upon which the AF WP StressKit plugin was installed by uploading the af-wp-stresskit.zip file using the plugin interface (Wittig and Wittig, 2023). The plugin has been applied and is running as can be confirmed through the Tools menu of the admin panel. In order to enter into the WordPress Admin panel, the default account name used was user and the password thereof had to be obtained through logs of the EC2 instance. This was achieved by first clicking on the instance on the AWS EC2 console after which the clock was clicked to open Actions > Monitor and troubleshoot > Get system log. In the system log, one line having a starting point of the words Setting Bitnami application password to: was found (Anand and Sharma 2022)The password that was shown in that line was pasted and was used to log in to /wp-admin. Through this, user could gain access credentials that were secure and one time and that could do this without any manual configuration during deployment, meaning it would give us immediate administration authority of the WordPress site.

2.3 Initial Testing

Figure 6: Displaying error message like Stress load: 0 (disabled)

Content management functionality was tested on two created blog posts. In AF WP StressKit, the internal stress factor was first fixed at zero to ensure the correct functioning. It was then changed to 500 and then to 60 which enables incremental simulation of the load on the server and monitoring latency in blog page refresh. By adjusting the value of stress factor to 0 in the AF WP StressKit plugin the message, Stress load: 0 (disabled), appeared at the bottom of the page on the website (Li et al. 2021).. It meant that there was no artificial load applied to the server and it also meant that the plugin was installed correctly as well as without any errors in its idle condition.

2.4 Challenges Encountered

Figure 7: Plugin activated

In the process of SSH access, the connection was initially blocked by firewall restrictions but the problem was solved by changing the source IP in the security group settings. As well, an upload of a plugin once failed because of a momentary break in session and the case was rectified by simply terminating the browser and re-upping the same file (Destarina et al., 2024).

3. Task B: Design and Implementation of Auto-Scaling

3.1 Analysis of Performance Bottlenecks

In an initial load test carried out with the AF WP StressKit, scenes of high latency set in when the number of concurrent users surpassed around 250 clients. The loading of the pages exceeded five seconds, which revealed that the EC2 instance was not able to manage high traffic. Since the WordPress configuration was on a single t2.micro instance, redundancy and backups were not allowed, and its performance was reduced when the instance's peak CPU consumption was reached (Manurung and Lubis, 2024). Such a boundary of vertical scalability showed that this sort of solution had to be horizontally scalable to guarantee responsiveness and availability.

3.2 Auto-Scaling Architecture

Figure 8: AMI selection

The limitations included incorporating Auto Scaling Group (ASG) based on a launch template based on a custom AMI composed of installed WordPress EC2 instances. This template also worked so every new case deployed had the WordPress environment and AF WP StressKit plugin installed to maintain consistency in the configuration. An application load balancer (ALB) was created to redirect traffic between alive EC2 instances in the ASG (Catillo et al., 2021). The ALB was also set up with a target group where the instances were registered dynamically according to health checks so that all the traffic could be distributed and failover in any one instance could be done. The ASG parameters were set to a minimum of 1 instance, desired one instance and the maximum number set to 3 instances.

3.3 CloudWatch Metric Selection

Its decision variable auto-scaling was calculated on the track of the CloudWatch metric CPU utilisation that was observed during the stress testing. The use of the CPU was constantly peaking at more than 75 per cent during the load. Hence, the scale-out policy was set to occur in cases where the overall average CPU usage was 70 per cent or more within two consecutive evaluation periods. The policy was set to eliminate an instance when average CPU usage remained low below 30 per cent over a long period in the case of scale-in (Juncosa Palahí, 2022). These limits were put in place to make sure that resources were scaled up appropriately and resources were not prematurely stopped.

3.4 Security and Reliability Considerations

The reason was to enable the ASG to work with EC2 instances and the EC2 instances to securely communicate with other AWS services like CloudWatch to which IAM roles were attached. The ASG was also set up across several availability zones in the same region to enhance resiliency. The policies to track down the targets were applied to keep the average CPU utilization at approximately 50 per cent to optimize responsiveness against cost-related efficiency and to stabilize the performance in wide load variability.

4. Task C: Load Testing and Optimization

4.1 Load Testing Setup

Figure 9: Target verification

Loader.io was picked as the main load testing tool since it is easy to integrate and has freemium access. Once the account is registered, the target host uses the EC2 public DNS of the WordPress application behind the Application Load Balancer. A verification token offered by loader.io was uploaded to the web server using SSH in order to confirm the control of the host. Through the terminal access code, the verification string was copied into a .txt file in the WordPress root directory by echo command (Leocadio, 2025). The host was appropriately verified by loader.io, which allowed configuring a load test on 500 clients in a 1-minute test frame. The verification of the target was the most important procedure so that the load testing could be provided with loader.io. loader.io then gave a verification token after entering the EC2 public DNS as the target host. This token was inserted in a.txt file and this file was uploaded to the root directory of WordPress through the SSH (Greenwald et al. 2022). After being accessed with the browser via a specific URL shared by the loader.io, the platform confirmed the control of the server. A successful verification enabled load tests to be set and run test on the deployed WordPress application.

4.2 Load Testing Scenarios

The load test that was done on the first was 250 clients within one minute. The system was able to handle the load with an average response time equal to circa 1.6 seconds and instance scaling was not triggered because the CPU was not utilized in excess of 70 percent. In the second scenario, a simulation of concurrent users was involved and there were 350 concurrent users. It caused the delay of the reply time to 2.4 sec, and the Auto Scaling Group triggered a scale-out event that started the second EC2 instance. The third and final test with 500 clients lasted the longest time and it reached over five seconds initially (Zarkeshmoghadam, 2024). The third instance was just started automatically, and the average latency, after thirty seconds, stabilized at the level of about 1.2 seconds, which means that the infrastructure managed to scale to accept the additional workload.

4.3 Observations and Results

Every test generated a collection of latency graphs and logs and revealed real-time response statistics and system behaviour. CloudWatch records supported that scaling was happening and as user could see there was a marked spike in CPU utilization which activated the auto-scaling ramp. Screenshots of the change in each EC2 instance's state and load test results were taken. The response graphs confirmed the fact that system performance was enhanced tremendously with every further instance of it being brought online (Sharma, 2023). Logs also indicated that when the load fell the scale-in policy terminated surplus instances following some periods of cooldown and the system was capable of returning it to its base status.

4.4 Optimization Strategies

The original cooldown period was 300 seconds but it was later on changed to 180 seconds in order to achieve faster scale-in response. In tests, it was able to determined that the instance type switch of t2.micro to t3.micro slightly quickened the response time at the scale but the price per hour was raised. The balance was reached when the t3.micro instances were introduced in high-traffic simulation and they were changed back to t2.micro in running the baseline. This enabled maximization of performance related to operational costs in disproportional ways.

References

Journals

• Catillo, M., Ocone, L., Villano, U. and Rak, M., 2021. Black-box load testing to support auto-scaling web applications in the cloud. International Journal of Grid and Utility Computing, 12(2), pp.139-148.
• Chauhan, S. and Arora, D., 2023, May. Development of Cloud based Smart Voice Assistant using Amazon Web Services. In 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC) (pp. 1217-1221). IEEE.
• Destarina, N., Negara, E.S., Supratman, E. and Ulfa, M., 2024. Implementation Docker and Kubernetes Scaling Using Horizontal Scaler Method for Wordpress Services. Sinkron: jurnal dan penelitian teknik informatika, 8(4), pp.2192-2196.
• Juncosa Palahí, M., 2022. Platform for deploying a highly available, secure and scalable web hosting architecture to the AWS cloud with Terraform (Bachelor's thesis, Universitat Politècnica de Catalunya).
• Kumar, L., Yadav, K., Singh, J. and Tripathi, K., 2021. AI-based voice assistance using AWS. International Journal of Innovative Research in Computer Science Technology (IJIRCST), 9(3), pp.82-87.
• Leocadio, P.H., 2025. AWS Master Class Chapter 02: Compute. Authorea Preprints.
• Manurung, M.G. and Lubis, A., 2024. Implementasi High-Availability WordPress Deployment Berbasis Teknologi AWS. Bulletin of Computer Science Research, 4(2), pp.162-169.
• Serra, G., Fara, P. and Casini, D., 2023, September. Enhancing the Availability of Web Services in the IoT-to-Edge-to-Cloud Compute Continuum: A WordPress Case Study. In 2023 26th Euromicro Conference on Digital System Design (DSD) (pp. 602-609). IEEE.
• Sharma, P., 2023. Moving production apps to AWS cloud and backup strategies.
• Wittig, A. and Wittig, M., 2023. Amazon Web Services in Action: An in-depth guide to AWS. Simon and Schuster.
• Yashina, E., Rubanik, T. and Chukhray, A., 2022. Speech-To-Text Software Design for the High Education Learning. In ProfIT AI (pp. 44-56).
• Zarkeshmoghadam, B.H., 2024. The process of migrating web applications to the Microsoft Azure Cloud.
• Balayn, A., Lofi, C. and Houben, G.J., 2021. Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems. The VLDB Journal, 30(5), pp.739-768.
• Anand, S. and Sharma, S., 2022. Hybrid Cloud Approaches for Large-Scale Medicaid Data Engineering Using AWS and Hadoop. International Journal of Emerging Trends in Computer Science and Information Technology, 3(1), pp.20-28.
• Li, X., Luo, W., Yuan, M., Wang, J., Lu, J., Wang, J., Lü, J. and Zeng, J., 2021, April. Learning to optimize industry-scale dynamic pickup and delivery problems. In 2021 IEEE 37th international conference on data engineering (ICDE) (pp. 2511-2522). IEEE.
• Greenwald, N.F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Fullaway, C.C., McIntosh, B.J., Leow, K.X., Schwartz, M.S. and Pavelchek, C., 2022. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature biotechnology, 40(4), pp.555-565.