The following is a tale of how we discovered and fixed a variety of reliability and performance issues in our Node.js application that were caused by Node.js event loop lag.

We are Trigger.dev, a background job platform with no timeouts for TypeScript developers. Check us out on GitHub and give us a ⭐

At around 11pm local time (10pm UTC) on Thursday June 20, we were alerted to some issues in our production services powering the Trigger.dev cloud, as some of our dashboard/API instances started crashing.

We hopped on to our AWS production dashboard and saw that the CPU usage of our instances were really high and growing.

Combing through the logs on the crashed instances, we came across various errors, such as Prisma transaction timeouts:

Followed by a message from Prisma stating they were unable to find the schema.prisma file and an uncaught exception, leading to the process exiting:

We were puzzled by these errors, as the load on our primary database was consistently under 10% and the number of connections was healthy. We were also seeing other errors causing crashing, such as sending WebSocket messages to an already closed connection (which yes, throws an exception)

At the same time, we noticed that our network traffic was spiking, which was odd as we weren't seeing a corresponding increase in requests to our API:

Just past 1am local time, we deployed a couple things we had hoped would fix the issue (spoler: they did not). While waiting for the deploy to complete, I hit our metrics endpoint and was hoping to find some clues. I indeed did find something:

The deploy finished and the CPU usage dropped and the crashes stopped, so we were hopeful we had fixed the issue and went to bed.

The next morning, we woke up to the same issue. The CPU usage was high again, the database was steady, and the network traffic was high. We were back to square one.

We had a hunch that if we could get a better idea behind the spikes in network traffic, we could find the root cause of the issue. We had access to AWS Application Load Balancer logs, so we setup the ability to query them in Athena and started by looking at the IPs with the most usage:

That's 21GB of data used by a single IP address in a single day. We wrote another query to see what paths this IP was hitting and causing the high network traffic:

Most of the traffic was coming from just a few different paths, but they were all pointing to our v3 run trace view (just an example):

But the query above showed just a few paths, with thousands of requests each. We dug into the database and saw that these runs had 20k+ logs each, and our trace view lacked pagination. We also "live reload" the trace view as new logs come in, at most once per second.

So someone was triggering a v3 run, and viewing the trace view, as 20k+ logs came in, refreshing the entire page every second. This was causing the high network traffic, but doesn't explain the high CPU usage.

The next step to uncovering the root cause was to instrument the underlying code with some additional telemetry. Luckily we already have OpenTelemetry setup in our application, which sends traces to Baselime.io. After adding additional spans to the code we started seeing traces like this:

As you can see in that trace above, a large amount of time was being spent inside the createSpans span. After about 30 seconds of digging, it was pretty clear what the issue was:

Yup, that's a nested loop. We were iterating over 20k+ logs, and for each log, we were iterating over all the logs again to find the parent log (20k * 20k = 400m). In Big O notation, this is O(n^2), which is not great (it's really bad).

We fixed the issue by creating a map of logs by ID, and then iterating over the logs once to create the spans. We quickly deployed the fix (along with limiting live-reloading the trace view when there are less than 1000 logs) and the CPU usage dropped, the network traffic dropped, and the crashes stopped (for now).

With this one issue fixed, we returned our attention to the event loop lag we had discovered the night before. We had squashed one instance of it, but we had a feeling this wasn't the only place it was causing issues.

Before we dive into how we discovered and fixed the rest of our event loop lag issues, let's take a step back and explain what event loop lag is.

Node.js only has a small amount of threads of operation, with the event-loop belonging to the main thread. That means that for every request that comes in, it's processed in the main thread.

This allows Node.js services to run with much more limited memory and resource consumption, as long as the work being done on the main thread by each client is small.

In this way Node.js is a bit like your local coffee shop, with a single barista. As long as no-one orders a Double Venti Soy Quad Shot with Foam, the coffee shop can serve everyone quickly. But one person can slow the whole process down and cause a lot of unhappy customers in line.

So how can this possibly scale? Well, Node.js actually will offload some work to other threads:

Reading and writing files (libuv) happens on a worker thread

IO operations (like database queries) are done on worker threads

Crypto and zlib, while CPU heavy, are also offloaded to worker threads

The beauty of this setup is you don't have to do anything to take advantage of this. Node.js will automatically offload work to worker threads when it can.

The one thing Node.js asks of you, the application developer, is to fairly distribute main-thread work between clients. If you don't, you can run into event loop lag, leading to a long queue of angry clients waiting for their coffee.

Luckily, the Node.js team has published a pretty comprehensive guide on how to avoid event loop lag, which you can find here.

After deploying the fix for the nested loop issue, we wanted to get a better idea of how often we were running into event loop lag. We added an event loop monitor to our application that would produce a span in OpenTelemetry if the event loop was blocked for more than 100ms:

The code above makes use of node:async_hooks to capture and measure the time spent between the before and after hooks. If the duration between the two is greater than 100ms, we create a span in OpenTelemetry, making sure to set the startTime to the time the before hook executed.

We deployed the monitor on Friday afternoon and waited for the results. Around 10pm local time, we started seeing spans being created with pretty large durations:

Starting the morning of Monday the 24th, we started on a PR to fix as many of the event loop lag issues as we could. We submitted and deployed the PR around 10am UTC on Thursday the 27th with the following fixes:

Added a limit of 25k logs to the trace view (after discovering a run with 222k logs). Added a "download all logs" button to allow users to download all logs if they needed them.

Added pagination to our v2 schedule trigger list after we saw a 15s lag when a user had 8k+ schedules.

Added a hard limit of 3MB on the input to a v2 task (after realizing we didn't have a limit at all), which matches the output limit on v2 tasks.

Importantly, we changed how we calculated the size of the input and output of a v2 task. Previously, we were using Buffer.byteLength after having parsed the JSON of the request body. This was causing the event loop to block while calculating the size of the input and output. We now calculate the size of the input and output before parsing the JSON, using the Content-Length header of the request.

We use Zod to parse incoming JSON request bodies, and removing the keys for the input and output of v2 tasks and handling them separately.

After deploying these fixes, we saw a significant drop in the number of spans being created by the event loop monitor:

We now have alerts setup to notify us if any event loop lag over 1s is detected, so we're able to catch and fix any new issues that arise.

While most of the event-loop lag issues have been fixed, we still have more work to do in this area, especially around payload and output sizes of tasks in v3. We already support much larger payloads and outputs in v3 as we don't store payloads/outputs larger than 512KB in the database, but instead we store them in an object store (Cloudflare R2).

But v3 payloads are still parsed in our main thread, but in an upcoming update we're going to be doing 2 things to help with this:

More importantly, we now have a much better understanding of what it takes to run a reliable Node.js service in production that can handle a large number of clients. Going forward, we're going to be more careful in designing our code to avoid event loop lag.

Trigger.dev is implemented as a long-lived Node.js process, which as you can see above, takes some additional work to ensure that a single client can't block the event loop for everyone else. This is a common issue with long-lived Node.js processes, and is one of the reasons why serverless is so popular.

But serverless isn't a silver bullet. While serverless functions are short-lived, they still run on a single thread, and can still run into event loop lag. This is especially true if you're using a serverless function to handle a large number of clients, or if you're doing a lot of synchronous work in your function.

To keep request latency low, function instances are kept warm for a period of time after they're booted (known as a cold-boot). This means that while this event loop lag issue is mitigated somewhat by having many more instances running, it's still something you need to be aware of when building serverless Node.js applications. You can read more about this issue here.

If you are running a long-lived Node.js service, requests aren't the only source of event loop lag that you need to be aware of. Offloading long-running or queued tasks inside the same Node.js process can also cause event loop lag, which will slow down requests and cause issues like the ones we experienced.

This is one reason why the pattern of using serverless for background jobs is so popular. By running background jobs in a separate process, you can avoid event loop lag issues in your main service and scale up and down to handle changing workloads. And serverless can be a great fit, but still requires careful design to avoid event loop lag and timeouts.

Trigger.dev v3 is now in Developer Preview and takes an approach that combines the best of both worlds. We deploy and run your background task code in a separate process, completely isolated from your main service and from each other, without worrying about timeouts. If you'd like to try it out, please sign up for the waitlist.

How we tamed Node.js event loop lag: a deepdive

MATLAB short for Matrix Laboratory is a programming language developed by
MathWorks. It is used for technical computing and is widely used in engineering,
science, and mathematics. MATLAB is a powerful tool that can be used for a
variety of tasks, including:

To set up MATLAB, you will need to download the software from the MathWorks
website. Once you have downloaded the software, you will need to install it on
your computer. The installation process is straightforward and should take a few
minutes to complete.

Once MATLAB is installed, you can start using it. The MATLAB interface is
divided into two main parts: the Command Window and the Editor.

The Command Window is where you type in MATLAB commands.

The Editor is where you write MATLAB scripts.

To learn more about using MATLAB, you can consult the MATLAB documentation or
take one of the many online tutorials that are available.

help: Displays help for a MATLAB command.

whos: Displays a list of all variables in the current workspace.

exp: Computes the exponential of a number.

sqrt: Computes the square root of a number.

max: Computes the maximum value of a set of numbers.

min: Computes the minimum value of a set of numbers.

sum: Computes the sum of a set of numbers.

mean: Computes the mean of a set of numbers.

std: Computes the standard deviation of a set of numbers.

MATLAB is a powerful tool that can be used for a variety of tasks. It is easy to
learn and use, and there are many resources available to help you get started.
If you are looking for a powerful tool for technical computing, MATLAB is a
great option.

Matlab

User Interface (UI) and User Experience (UX) design play a crucial role in shaping the digital experiences of users. By focusing on usability, accessibility, and aesthetics, UI/UX designers create interfaces that are not only visually appealing but also intuitive and engaging.

UI/UX design begins with understanding the needs, preferences, and behaviors of users. By conducting user research, creating personas, and mapping user journeys, designers gain valuable insights into how to design interfaces that meet user expectations and address pain points effectively.

Consistency in design elements such as layout, color scheme, typography, and navigation enhances usability and promotes a sense of familiarity for users. By maintaining consistent design patterns throughout an interface, designers make it easier for users to navigate and interact with the product.

Designing for accessibility ensures that digital products are usable by people of all abilities. Considerations such as color contrast, text legibility, keyboard navigation, and screen reader compatibility are essential for creating inclusive experiences that cater to a diverse audience.

4. Visual Hierarchy and Information Architecture

Visual hierarchy guides users' attention and helps them prioritize information on a screen. By organizing content into clear hierarchies and using visual cues such as size, color, and contrast, designers can improve the readability and comprehension of interfaces.

Providing feedback to users is essential for creating responsive and interactive interfaces. Visual cues such as animations, transitions, and micro-interactions inform users of their actions and help them understand the system's response, enhancing the overall user experience.

UI/UX design is a collaborative process that involves understanding user needs, applying design principles, and iterating on solutions based on feedback. By prioritizing usability, accessibility, and aesthetics, designers can create user-centric experiences that delight and engage users across digital platforms.

For further reading on UI/UX design principles and best practices, check out the following resources:

The Design of Everyday Things by Don Norman

Crafting User-Centric Experiences: A Guide to UI/UX Design Principles

In the ever-evolving landscape of web development, Progressive Web Applications (PWAs) have emerged as a game-changer, offering users a seamless and engaging experience akin to native mobile apps. This article delves into the rise of PWAs and their transformative impact on user experience (UX).

Understanding Progressive Web Applications (PWAs)

Progressive Web Applications are web-based applications that leverage modern web technologies to provide a native app-like experience to users. Unlike traditional websites, PWAs offer features such as offline functionality, push notifications, and access to device hardware APIs, blurring the line between web and native apps.

: PWAs are designed to work reliably, even in poor network conditions, thanks to features like service workers that cache content and enable offline access.

: PWAs deliver fast and smooth performance, with optimized loading times and smooth animations, enhancing the overall user experience.

: With features like push notifications and home screen installation prompts, PWAs encourage user engagement and retention, similar to native apps.

PWAs have revolutionized the way users interact with web applications, offering a host of benefits that enhance the overall user experience.

PWAs are known for their fast loading times and smooth performance, even on low-end devices or slow network connections. This improved performance translates to a more enjoyable and frictionless user experience, leading to higher user satisfaction and engagement.

One of the most significant advantages of PWAs is their ability to work offline. By leveraging service workers and caching strategies, PWAs can store content locally, allowing users to access app features and content even without an internet connection. This offline functionality is particularly useful for users in areas with limited connectivity or during network outages.

PWAs can be installed directly onto a user's device, bypassing the need for app store downloads and installations. With features like the "Add to Home Screen" prompt, users can easily add PWAs to their device's home screen, creating a native app-like experience. This seamless installation process improves accessibility and encourages users to engage with PWAs regularly.

PWAs are built using web technologies like HTML, CSS, and JavaScript, making them compatible across different platforms and devices. Whether accessed on a desktop, smartphone, or tablet, PWAs provide a consistent user experience, eliminating the need for separate development efforts for each platform.

As the demand for fast, reliable, and engaging web experiences continues to grow, PWAs are poised to play an increasingly significant role in the future of web development. With advancements in web technologies and browser support for PWA features, we can expect to see even more innovative and immersive web applications that rival the capabilities of native mobile apps.

Progressive Web Applications have redefined the way users interact with web content, offering a native app-like experience with the accessibility and reach of the web. With their enhanced performance, offline functionality, and seamless installation process, PWAs have raised the bar for user experience on the web. As businesses and developers embrace PWAs, we can anticipate a future where web applications blur the lines between web and native experiences, providing users with a truly immersive and engaging digital environment.

The Rise of Progressive Web Applications (PWAs) and Their Impact on User Experience

In recent years, influencer marketing has become an integral part of e-commerce strategies, offering brands a unique opportunity to leverage the social influence of online personalities to promote their products and services.

The Impact of Influencer Marketing on E-commerce

Influencer marketing offers several benefits for e-commerce businesses, including:

: Collaborating with influencers allows brands to tap into their followers' networks, expanding their reach and generating brand awareness among new audiences.

: Influencers often have a loyal and engaged following who trust their recommendations. By partnering with influencers who align with their brand values, e-commerce businesses can build credibility and trust with consumers.

: Influencers have the ability to target specific demographics and niche markets, making influencer marketing an effective strategy for reaching highly relevant audiences.

Best Practices for Successful Influencer Marketing Campaigns

To harness the power of influencer marketing effectively, e-commerce businesses should consider the following best practices:

: Choose influencers whose values, interests, and audience demographics align with your brand. Look for influencers with an engaged and authentic following.

: Define specific goals for your influencer marketing campaign, whether it's increasing brand awareness, driving website traffic, or boosting sales. Establish key performance indicators (KPIs) to measure the success of your campaign.

: Work with influencers to develop creative and engaging content that resonates with their audience while showcasing your products or services in an authentic way.

: Monitor the performance of your influencer marketing campaigns using analytics tools to track key metrics such as reach, engagement, and conversion rates. Use this data to optimize future campaigns and maximize ROI.

Influencer marketing has emerged as a powerful strategy for e-commerce businesses looking to stand out in a crowded marketplace. By harnessing the social influence of online personalities, brands can connect with their target audience in a more authentic and meaningful way, driving engagement, loyalty, and ultimately, sales.

The Rise of Influencer Marketing in E-commerce: Leveraging Social Influence for Brand Growth

The fields of Computer Science (CS) and Information Technology (IT) are continuously evolving, driven by groundbreaking innovations that promise to revolutionize various aspects of our lives.

Artificial Intelligence (AI) and Machine Learning (ML)

AI and ML technologies are at the forefront of innovation in CS and IT. From autonomous vehicles to personalized healthcare solutions, AI and ML algorithms are being deployed to tackle complex problems and enhance efficiency across industries.

Recent advancements in deep learning algorithms have led to significant breakthroughs in natural language processing, computer vision, and predictive analytics, paving the way for smarter, more autonomous systems.

With the proliferation of digital technologies, cybersecurity has become a top priority for organizations worldwide. CS and IT professionals are developing advanced cybersecurity solutions to protect sensitive data and defend against cyber threats.

From blockchain-based authentication mechanisms to AI-driven threat detection systems, the cybersecurity landscape is constantly evolving to stay ahead of sophisticated cyber attacks and safeguard critical infrastructure.

Cloud computing and edge computing technologies are transforming the way data is stored, processed, and accessed. By leveraging the scalability and flexibility of cloud infrastructure, organizations can deploy applications and services more efficiently and cost-effectively.

Meanwhile, edge computing solutions are bringing computing power closer to the source of data generation, enabling real-time processing and analysis for applications such as IoT devices, autonomous vehicles, and smart cities.

As we look to the future, the potential impact of advancements in CS and IT cannot be overstated. From AI-powered innovations to cybersecurity solutions and cloud computing technologies, the possibilities are endless.

By staying abreast of the latest developments and embracing emerging technologies, businesses and individuals alike can harness the power of CS and IT to drive innovation, improve productivity, and address some of the most pressing challenges of our time.

From artificial intelligence to cybersecurity, the fields of Computer Science and Information Technology are experiencing rapid advancements that are reshaping industries and transforming the way we live and work.


Exploring the Latest Innovations in Computer Science and Information Technology

Typography, the art and technique of arranging type to make written language legible, readable, and appealing, has a fascinating history that spans thousands of years. From the earliest forms of written communication to the digital fonts used in today's digital age, typography has evolved significantly, reflecting changes in technology, culture, and design trends.

The history of typography can be traced back to ancient civilizations, where written communication first emerged. In Mesopotamia, around 3500 BCE, the Sumerians developed one of the earliest writing systems known as cuneiform, consisting of wedge-shaped characters carved into clay tablets. Similarly, in ancient Egypt, hieroglyphs were used for inscriptions on temples, tombs, and monuments.

: The Phoenicians developed the first alphabetic writing system around 1050 BCE, consisting of 22 consonant letters. This innovation laid the foundation for the alphabets used in many modern languages.

: In 1440, Johannes Gutenberg invented the movable type printing press, revolutionizing the production of books and facilitating the spread of knowledge during the Renaissance. Gutenberg's printing press marked the beginning of the modern era of typography.

The invention of printing technology in the 15th century led to a flourishing of typographic design during the Renaissance and Baroque periods. Printers and typographers such as Claude Garamond, William Caslon, and John Baskerville created iconic typefaces that are still used today.

: During the 18th and 19th centuries, transitional and modern typefaces emerged, characterized by refined serifs, vertical stress, and contrast between thick and thin strokes. Examples include Baskerville, Bodoni, and Didot.

: The Industrial Revolution in the 19th century brought innovations in printing technology and mass production, leading to the proliferation of newspapers, posters, and advertisements. Sans-serif typefaces like Helvetica and Futura became popular for their clean and modern aesthetic.

The advent of computers and digital technology in the 20th century revolutionized typography once again, paving the way for the creation of digital fonts and desktop publishing software. Designers and typographers now had access to a vast array of typefaces and tools for creating and manipulating text.

: TrueType and OpenType font formats were introduced in the late 20th century, offering greater flexibility and compatibility across different operating systems and software applications.

: With the rise of the internet and mobile devices, responsive web typography has become essential for ensuring optimal readability and legibility across various screen sizes and resolutions.

Typography has come a long way since its humble beginnings in ancient inscriptions. From the invention of movable type printing to the digital fonts of the 21st century, typography continues to evolve, driven by advancements in technology and design innovation. Understanding the rich history of typography helps designers appreciate its significance and influence on visual communication in the modern world.

Blogs