OPTIMIZATION FOR SPECULATIVE EXECUTION IN BIGDATA PROCESSING CLUSTERS project in Java Spring Framework.

A big parallel processing job can be delayed substantially as long as one of its many tasks is being assigned to an unreliableor congested machine. To tackle this so-called straggler problem, most parallel processing frameworks such as MapReduce haveadopted various strategies under which the system may speculatively launch additional copies of the same task if its progress isabnormally slow when extra idling resource is available. In this paper, we focus on the design of speculative execution schemes forparallel processing clusters from an optimization perspective under different loading conditions. For the lightly loaded case, we analyzeand propose one cloning scheme, namely, the Smart Cloning Algorithm (SCA) which is based on maximizing the overall system utility.Wealso derive the workload threshold under which SCA should be used for speculative execution. For the heavily loaded case, we proposethe Enhanced Speculative Execution (ESE) algorithm which is an extension of the Microsoft Mantri scheme to mitigate stragglers. Oursimulation results show SCA reduces the total job flowtime, i.e., the job delay/ response time by nearly 6% comparing to the speculativeexecution strategy of Microsoft Mantri. In addition, we show that the ESE Algorithm outperforms the Mantri baseline scheme by 71% interms of the job flowtime while consuming the same amount of computation resource

Java Development Kit (JDK) Version: Java 8 or higher (Java 11 or Java 17 is recommended as of 2024 for LTS support). Spring Framework IDE: Eclipse, IntelliJ IDEA, or VS Code with Java and Spring Boot extensions. Database: MySQL, PostgreSQL or MongoDB. Application Server : Need a web server like Apache Tomcat or Jetty.

Processor (CPU): Minimum: 2-core processor (e.g., Intel Core i3 or equivalent) Recommended: 4-core or better (e.g., Intel Core i5/i7, AMD Ryzen 5/7) RAM (Memory): Minimum: 8 GB of RAM Recommended: 16 GB of RAM (especially if running Docker, multiple services, or heavy IDEs) Hard Disk: Minimum: 50 GB of free disk space (SSD is recommended for faster performance) Recommended: 100 GB or more (for large projects or multiple repositories) Display: Minimum: 1080p resolution (1920x1080) for better productivity, especially if using modern IDEs with multiple panes (for coding, testing, etc.) Recommended: Dual monitors for multitasking (optional but helpful) Operating System: Windows 10/11, macOS, or Linux (Ubuntu, CentOS, etc.) Linux is often preferred for server-side development due to its compatibility with Spring and deployment environments.