5004CMD Big Data Programming Project Sample

1. Introduction

Transportation analytics is now more critical for designing, improving and articulating urban growth, transport infrastructures, para-transit infrastructures and lastly for decisions that encompass health of citizens in the societies. Mobile devices and tracking tools are becoming increasingly popular and therefore heavy amounts of mobility data is now available. But, a crucial factor that yet hinders the majority of organizations, including the government departments is the effectiveness in processing large datasets. In this project, a real-world problem proposed by the Bureau of Transportation Statistics (BTS) is solved, which carries out a new analysis for the mobility statistic that has some practical difficulties because of computational complexity nowadays.

1.1 Problem Description

The amount of trips is simplified by the Bureau of Transportation Statistics (BTS) under its mobility statistics program. It turned out that downloading current analytical processes is very time-consuming and therefore they cannot afford to do frequent analyses on these rather large amounts of data. This called for a solution that was:

• Accomplish a large amount of work with a large amount of trips data.
• Identify trends in home-staying and traveling.
• Categorize trips by distance.
• Determine cyclic trends of travel patterns.
• Establish the accuracy of travel frequencies based on several factors.

The task here is that these analyses should be done using parallel processing which is much faster than using the sequential processing methods.

1.2 Project Aim

The project includes analyzing the US mobility data by parallel processing involving home stays and travels, calculating the travelled distance, comparing the results received by parallel and serial processing, as well as creating the model that predicts the frequency of travelling depending on the length of the trip.

1.3 Purpose of Research Project

The focus of this work is to show how big data analytics and parallel computing can help to change current processes in the field of transportation data analysis for the needs of authorities. This means that the BTS can conduct more analyses in less time, as the latter is of great importance in policy making and planning. It is also useful in planning of infrastructure, health and economic planning since the patterns of travel can be analyzed.

1.4 Project Objectives

The following are the specific objectives that will be achieved by the end of this project:

• To compare the average general population of people that are quarantined at home per week.
• To work out travel distances for those who are not restricted to their home.
• To determine specific trip categories that prevail at some particular dates.
• To assess the relative efficiency of parallel processing on the basis of 10 and 20 processors.
• To build up a frequency trip model with an intent of targeting the extent and/or trip length.
• To perform exploratory spatial data analysis and represent the travel data on the map so as to express a clear message.

1.5 Hypothesis

Based on the above-discussed literature, the following hypotheses are developed for this research.

• Dask planning is an improvement on the basic pandas plan and allows for parallels in the form of using multiple processes and is much more efficient as the number of processors increases.
• Distance and frequency of traveling depend upon the demography of the population and the latter can be estimated using regression analysis.
• People also travel for other purposes, and the distance differs with many of the travel distances being short to medium range.

2. Project Methodology

2.1 Data Acquisition

For the purpose of the present analysis, two basic datasets were employed:

• ‘Trips_by_Distance.csv’: This file consists of data regarding the number of trips made and the distance covered by the travelers in different ranges.
• ‘Trips_Full_Data.csv’: Contains comprehensive trip data with additional metrics

The datasets were first loaded into pandas to consider the basic statistics of the data and then loaded into dask for multiprocessing (Parker et al. 2021). These datasets include the level of social mobility, which is summarized as the number home and the number of trips by distance class, plus temporal data in forms of date, week, month.

2.2 Data Pre-processing

The data needed some amount of pre-processing to bring the data to a format that is fit for further analysis and use.

• The date fields were thus converted to datetime formats for better analysis in terms of time (Stiles & Smart, 2021).
• Missing values were presented and managed to avoid biases within the analyses.

These preprocessing allowed the data to be in the right format that would adequately prepare it for the subsequent analysis and-modeling.

2.3 Data Cleaning

In data cleaning there were issues of missing observations, inconsistent records, and data entries that might be duplicates. That is why, when examining the datasets, some null values were noted that needed to be filled. These were filled with zeros as required by the analysis done earlier (Politis et al. 2021). This approach was chosen because any value of zero in this case means no trips or no people, and it is not ‘missing’ data in the strictest definition of the term.

2.4 Data Categorization

The data was then the analyzed in a manner that will meet the research objectives based on the following classifications:

• Time segmentation analysis: Data was collected in a weekly format to be able to compare results over a particular period of time (Škare, Soriano, & Porada-Rochoń, 2021).
• Distance-based categorization: The trips were categorized in different distance groups.

These categorizations enabled more refined studies of certain aspects of the travel information, mainly the temporal and spatial differences in accessing travel.

2.5 Parallel Computing

Dask was used in parallelism to increase the speed of processing large datasets. The processing time was used to measure the computational efficiency for each of the configurations under consideration (Patwary & Khattak, 2024). This study provided evidence that parallel processing shortened the computing time than sequential processing, but the increase from 10 to 20 processors represented a declining curve.

2.6 Data Processing

There were several analytical tasks done in the data processing stage, including

• The calculation of the average number of people staying at home per week.
• Taking distances traveled by persons who are not housebound in a given day (Doborjeh et al. 2022).
• Identifying specific trip categories that translate to high numbers of dates.

All these operations have been executed in sequential mode (pandas) and parallel mode (dask) for benchmarking.

2.7 Data Fitting - Model Selection

Linear regression approach was used to establish the relationship, which connects the number of people staying at home and the number of trips made by car in certain categories of distances. Linear regression was selected as it is easy to understand and suitable to find the linear patterns extracted during the exploratory data analysis.

2.8 Model Testing

The model was also assessed using train-test split technique to determine the ability of the model in making accurate predictions. The accuracy of the model was assessed by recognizing Mean Squared Error (MSE) into consideration as well as the coefficient of determination (R²) (Wang et al. 2023). The obtained ‘R-squared’ value is 0.862; this means that the model accounts for 86.2% of the variation of the number of trips implying good prediction.

2.9 Data Analysis

The following depicted an analysis of the research objectives as articulated at the beginning of this research:

• The weekly breakdown of the ‘Population Staying at Home’ variable also indicated that the level of home-staying was not stagnant and was evolving with time (Feng et al. 2021).
• The frequency analysis of the trips by distance limit indicated that many of the tripping activities were primarily constituted by short to moderately long distance.
• The temporal pattern of travel behavior was observed by analyzing dates with more than 10 million trips in the given distinctive distance segmentation (Moro et al. 2021).
• The analysis of the processing time showed that parallel processing provided advantages over the computational approaches.

These analyses gave a deep understanding of the traveling patterns and computation measure.

2.10 Data Visualization

Some of the methods used to map the results in figures include:

• Graphs for illustrating the number of people at home and on the roads.
• Line graphs for portraying the number of trips against dates (Fotiadis, Polyzos, & Huan, 2021).
• Production of bar charts for comparing processing times.
• Visualization of model predictions (Dingil & Esztergár-Kiss, 2021).

These visualizations were able to convey all the patterns and interconnections found in the data well.

3. Results

3.1 Parallel Processing

This research also revealed that the processing time difference between 10 and 20 processors was not significantly different with the increase in the number of processors.

Figure 1: Parallel Computing

This implies that for the given dataset size and operations in this study, the number of processors should be maximum 10 and beyond this, increasing the number of processors will not yield significant improvement. This is presumably due to distribution of the load and consolidation of results in groups of numbers that are significant.

3.2 Data Visualization

Figure 2: People Staying at Home vs Travelling

The daily number of person-statics and the distribution of people staying at home against week displayed differences in the trend, and the degree of increase in later week 44-52 may be due to certain events or seasonal factors.

Figure 3: Number of Trips by Date

They also demonstrated that most units are in the categories of travel distance up to or equal to 25 miles with relatively fewer units in the distance bands above 25 miles. Although not wholly, this pattern resembles other normal city and suburban travel habits.

Figure 4: Computational Efficiency

Although there were trips that occurred consistently across distance categories, there were special dates characterized by an increased number of trips. This indicates that there is a pattern in the movement of people and it could be occasion-related or time of the year related.

3.3 Data Analysis

The movement by people at home was relatively constant in the initial weeks and a sudden spike was recorded between weeks 44 to 52. This pattern may reflect seasonal variations or changes, holidays, or some other special circumstances.

Figure 5: Linear Regression

The model used to forecast the number of trips with the population that remains at home, presented satisfactory results with the coefficient of determination equal to 0.862. The calculated mean squared error of 1,906,572,453,992.013 can be considered as sufficiently large in an absolute scale as long as it is adequate in the given large size of the data set.

This comparison of model prediction with actual data of the corresponding scenario represented in figure 8 above is quite good with a general appearance of the same pattern with a sense of slight variation at the higher and lower end respectively.

4. Discussion

4.1 Data Communication

The data communication used in this project involved presenting complex mobility patterns in a simplified manner. Histograms, the scatter plots, and the model used in the paper allowed for explaining the complexity of the travel behavior (Borkowski, Jażdżewska-Gutta, & Szmelter-Jarosz, 2021). In this case, the parallel processing comparison was done through the direct measurement of time as well as use of bar charts that enhanced the display of the efficiency of parallel computing.

4.2 Data Interpretation

The factor of people staying at their homes in weeks 44-52 could be attributed to seasonal or particular events leading to the rise in communications. This pattern has to be explored further with a view of unravelling the factors that lead to the emergence of such complexes (Hunter et al. 2021). The positive correlation between the population staying at home and the number of trips in the 10-25 miles range implies that these variables have a causal link that might be mediated by some fundamental factors, for instance, the economic affluence, weather condition, or any public sanitation. Smoothed efficiency results of 10 and 20 processors indicate that there is an optimum limit above which further increase in parallel processing is not going to be much effective in case of this lot size (Hosseinzadeh et al. 2021). This point will be beneficial in determining the amount of computation resources that needs to be invested in subsequent analysis.

4.3 Future Recommendations

The following recommendation can be given based on the findings:

• Parallel processing: the circumstances denote that the utilization of about ten processors would be the most suitable for datasets of this magnitude. For future work, the study should seek to identify the optimal point at which the curve for different forms of the dataset and operations flattens.
• The fluctuations by week: There are large differences from week to week and possibly more specific temporal analyses, either at a day by day level or controlling for seasonal matters, could be beneficial (Jaipuria, Parida, & Ray, 2021).
• Model enhancement: Even though the analysis based on linear regression gave good results, furthering the model by extending the degree of polynomial regression or even developing a machine learning model can capture these nonlinear effects in a better way.
• Extension with data from outside: Addition of other variables like weather conditions, factors in the economy, health-wise indicators could improve the models’ performance (Barbieri et al. 2021).
• Interactive tools: For communicating such insights to the policy-makers, experts can design and construct interactive dashboards that the analysts can use to easily interpret the details depicted.

5. Project Management

The project was implemented based on the schedule and timeline and there is constant monitoring of progress and attainment of goals and objectives. It was necessary to acquire data and organize its preprocessing, implement the analysis, create models, and prepare the reports.

5.1 Risk Management

Besides, the following risks were implemented and controlled throughout the project:

• Data quality issues: The internal and external data quality issues are tackled through the conduct of a data cleansing and validation (Park et al. 2023).
• Challenges of computation: Addressed by using parallel processing algorithms in the execution of the models.
• Issue number five: performance of the developed models: controlled through testing and validation (Almlöf et al. 2021).
• Timeline constraints: Controlled through effective project planning and prioritization.

5.2 Gantt Chart

A detailed project plan in a Gantt chart spanning over 16 weeks from March to August proves useful in the execution of the project with each week corresponding to twelve successive phases that includes requirement analysis and data processing phase, parallel computing implementation phase, model development phase, and documentation delivery phase among others.

6. Conclusion

This project was able to effectively show how big data analytics and parallel processing can be employed in understanding the travel patterns of Americans. The latter proved to help to understand how many people tend to be in their homes rather than traveling and how far they travel and related tendencies change. The parallel processing done using dask proved to be faster than the sequential processing especially from the 10 processors up to 20 processors though the rate of improvement was not as fast as the previous one. Thus, this finding addresses issues of concern in the best possible way to allocate computational resources for transportation data analysis.

In developing the predictive model on the number of trips dependent on the population staying at home, the model yielded high accurate results giving out at 86.2% of the variance. This model can be beneficial to be used in the planning and formulation of transport policies. Some of the weaknesses of the study are the fact that some features of the data are not very detailed and specificity of certain constructs is not very high due to which study is limited to few variables. More research about temporal patterns and more comprehensive sets of variables and methodologies continue this research. In summary, this project highlighted the value of big data analytics when applied to the field of transportation – particularly its uses will be made more effective and accessible for policymakers.

Reference materials and sample papers are provided to clarify assignment structure and key learning outcomes. Through our Assignment Help UK, guidance is reflected while maintaining originality and ethical academic practice. The 5004CMD Big Data Programming Project Sample demonstrates practical application of big data analytics, parallel processing, and predictive modeling techniques.

References

Journals

• Almlöf, E., Rubensson, I., Cebecauer, M., & Jenelius, E. (2021). Who continued travelling by public transport during COVID-19? Socioeconomic factors explaining travel behaviour in Stockholm 2020 based on smart card data. European Transport Research Review, 13, 1-13. Retrieved from: https://link.springer.com/content/pdf/10.1186/s12544-021-00488-0.pdf [Retrieved on: 01.03.25]
• Barbieri, D. M., Lou, B., Passavanti, M., Hui, C., Hoff, I., Lessa, D. A., ... & Rashidi, T. H. (2021). Impact of COVID-19 pandemic on mobility in ten countries and associated perceived risk for all transport modes. PloS one, 16(2), e0245886. Retrieved from: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0245886&type=printable [Retrieved on: 01.03.25]
• Borkowski, P., Jażdżewska-Gutta, M., & Szmelter-Jarosz, A. (2021). Lockdowned: Everyday mobility changes in response to COVID-19. Journal of Transport Geography, 90, 102906. Retrieved from: https://pmc.ncbi.nlm.nih.gov/articles/PMC9188832/pdf/main.pdf [Retrieved on: 01.03.25]
• Dingil, A. E., & Esztergár-Kiss, D. (2021). The influence of the Covid-19 pandemic on mobility patterns: The first Wave’s results. Transportation Letters, 13(5-6), 434-446. Retrieved from: https://www.researchgate.net/profile/Ali-Dingil/publication/350021308_The_Influence_of_the_Covid-19_Pandemic_on_Mobility_Patterns_The_First_Wave's_Results/links/604bcc53a6fdcccfee79bed3/The-Influence-of-the-Covid-19-Pandemic-on-Mobility-Patterns-The-First-Waves-Results.pdf [Retrieved on: 01.03.25]
• Doborjeh, Z., Hemmington, N., Doborjeh, M., & Kasabov, N. (2022). Artificial intelligence: a systematic review of methods and applications in hospitality and tourism. International Journal of Contemporary Hospitality Management, 34(3), 1154-1176. Retrieved from: https://pure.ulster.ac.uk/files/98492847/Zohreh_et_al_AI_in_Tourism_Main_Manuscript_accepted_version_.pdf [Retrieved on: 01.03.25]
• Feng, L., Zhang, T., Wang, Q., Xie, Y., Peng, Z., Zheng, J., ... & Gao, G. F. (2021). Impact of COVID-19 outbreaks and interventions on influenza in China and the United States. Nature communications, 12(1), 3249. Retrieved from: https://www.nature.com/articles/s41467-021-23440-1.pdf [Retrieved on: 01.03.25]
• Fotiadis, A., Polyzos, S., & Huan, T. C. T. (2021). The good, the bad and the ugly on COVID-19 tourism recovery. Annals of tourism research, 87, 103117. Retrieved from: https://pmc.ncbi.nlm.nih.gov/articles/PMC7832145/pdf/main.pdf [Retrieved on: 01.03.25]
• Hosseinzadeh, A., Algomaiah, M., Kluger, R., & Li, Z. (2021). E-scooters and sustainability: Investigating the relationship between the density of E-scooter trips and characteristics of sustainable urban development. Sustainable cities and society, 66, 102624. Retrieved from: https://www.academia.edu/download/111435889/j.scs.2020.10262420240213-1-wd1kgf.pdf [Retrieved on: 01.03.25]
• Hunter, R. F., Garcia, L., de Sa, T. H., Zapata-Diomedi, B., Millett, C., Woodcock, J., ... & Moro, E. (2021). Effect of COVID-19 response policies on walking behavior in US cities. Nature communications, 12(1), 3652. Retrieved from: https://www.nature.com/articles/s41467-021-23937-9.pdf [Retrieved on: 01.03.25]
• Jaipuria, S., Parida, R., & Ray, P. (2021). The impact of COVID-19 on tourism sector in India. Tourism Recreation Research, 46(2), 245-260. Retrieved from: https://repository.londonmet.ac.uk/6219/1/Accepted-manuscript_with-authors-details.pdf [Retrieved on: 01.03.25]
• Moro, E., Calacci, D., Dong, X., & Pentland, A. (2021). Mobility patterns are associated with experienced income segregation in large US cities. Nature communications, 12(1), 4633. Retrieved from: https://www.nature.com/articles/s41467-021-24899-8.pdf [Retrieved on: 01.03.25]
• Park, K., Esfahani, H. N., Novack, V. L., Sheen, J., Hadayeghi, H., Song, Z., & Christensen, K. (2023). Impacts of disability on daily travel behaviour: A systematic review. Transport reviews, 43(2), 178-203. Retrieved from: https://drive.google.com/file/d/1Svbu3wccRd2ALHns0dHFPQx3v2Rg4hgp/view [Retrieved on: 01.03.25]
• Parker, M. E., Li, M., Bouzaghrane, M. A., Obeid, H., Hayes, D., Frick, K. T., ... & Chatman, D. G. (2021). Public transit use in the United States in the era of COVID-19: Transit riders’ travel behavior in the COVID-19 impact and recovery period. Transport policy, 111, 53-62. Retrieved from: https://www.sciencedirect.com/science/article/pii/S0967070X21002067 [Retrieved on: 01.03.25]
• Patwary, A. L., & Khattak, A. J. (2024). Interaction between information and communication technologies and travel behavior: using behavioral data to explore correlates of the COVID-19 pandemic. Transportation Research Record, 2678(12), 309-322. Retrieved from: https://pmc.ncbi.nlm.nih.gov/articles/PMC9396749/pdf/10.1177_03611981221116626.pdf [Retrieved on: 01.03.25]
• Politis, I., Georgiadis, G., Papadopoulos, E., Fyrogenis, I., Nikolaidou, A., Kopsacheilis, A., ... & Verani, E. (2021). COVID-19 lockdown measures and travel behavior: The case of Thessaloniki, Greece. Transportation Research Interdisciplinary Perspectives, 10, 100345. Retrieved from: https://www.sciencedirect.com/science/article/pii/S259019822100052X [Retrieved on: 01.03.25]
• Škare, M., Soriano, D. R., & Porada-Rochoń, M. (2021). Impact of COVID-19 on the travel and tourism industry. Technological forecasting and social change, 163, 120469. Retrieved from: https://pmc.ncbi.nlm.nih.gov/articles/PMC9189715/pdf/main.pdf [Retrieved on: 01.03.25]
• Stiles, J., & Smart, M. J. (2021). Working at home and elsewhere: daily work location, telework, and travel among United States knowledge workers. Transportation, 48(5), 2461-2491. Retrieved from: https://link.springer.com/content/pdf/10.1007/s11116-020-10136-6.pdf [Retrieved on: 01.03.25]
• Wang, K., Qian, X., Fitch, D. T., Lee, Y., Malik, J., & Circella, G. (2023). What travel modes do shared e-scooters displace? A review of recent research findings. Transport Reviews, 43(1), 5-31. Retrieved from: https://www.tandfonline.com/doi/pdf/10.1080/01441647.2021.2015639 [Retrieved on: 01.03.25]