Node kinda sucks on serverless...

Summiz Holo

Vercel's serverless advancements, multi-concurrency features, and interactive workload optimizations

• Servers are effective, leading Vercel to incorporate them into their serverless offerings.
• The discussion covers serverless strengths and weaknesses, as well as the workings of Node.
• Vercel's new multi-concurrency feature for their serverless functions is a noteworthy advancement.
• Multi-concurrency can significantly reduce compute usage and costs without impacting latency.
• Serverless products are optimized for interactive workloads like server rendering, APIs, and AI applications.
• Previous serverless models (like AWS Lambda) mapped one function instance to one invocation, contrasting with traditional Node server setups.

Lambda and serverless architectures with single request handling, cost implications of separate machine processing, inefficiencies in dynamic workloads, Node.js high concurrency capabilities, Forell functions enabling per instance concurrency, and IO handling impacts on server application efficiency

• Lambda and serverless architectures allow for handling requests without worrying about allocation, but only handle one request at a time per instance.
• In serverless environments, each request is processed on a separate machine, leading to increased costs, especially for long response times.
• The model struggles with interactive dynamic workloads, as single user requests are tied to one function instance, leading to inefficiencies.
• Node.js is designed for high concurrency and performs well with IO operations, which is often underestimated.
• Forell functions break away from Lambda's one-to-one mapping of invocation to compute, allowing for per instance concurrency, resulting in cost reductions.
• The handling of IO outside of the application logic can cause programs to halt, impacting efficiency in server applications.

CPU and memory operation inefficiencies, non-blocking software optimizations, Node.js event loop concurrency, serverless architecture resource waste, AWS Lambda cost implications, and enhanced serverless function capabilities

• There's a significant difference between CPU/memory operations and accessing external resources like disks or networks, which can lead to millions of clock cycles being wasted while waiting for responses.
• Better software can optimize performance by not blocking processes and allowing other tasks to proceed while waiting for database queries.
• The event loop concept in Node.js allows for high concurrency by enabling execution to continue while waiting for I/O operations, avoiding wasted CPU cycles.
• With serverless architecture, each request operates on its own instance, which can lead to inefficient use of resources when waiting for requests since other users cannot utilize those cycles.
• Node's built-in concurrency benefits are diminished in serverless environments like AWS Lambda, leading to increased costs and wasted cycles.
• The introduction of streaming support in Lambda and the development of a function invocation service by Vercel enhance the capabilities of serverless functions beyond traditional models.

Lambda function architecture with secure TCP streaming, bidirectional invocation, Rust performance enhancements, and cold start challenges

• They built a layer to control how the Lambda is used, enabling features like partial pre-rendering.
• The function opens a secure TCP socket for streaming responses, even if Lambda doesn't support streaming.
• Lambda has two invocation models: request invocation and manual invocation via an API.
• A bidirectional stream to all functions could enable additional work and functionality.
• Rust is involved in rewriting the function runtime for performance improvements and better debuggability.
• The runtime acts as a hypervisor that orchestrates workloads and ensures correct code execution.
• The new architecture includes edge routing, a function load balancer, and a function invocation service.
• The function invocation service can manage connections and serve multiple requests with a single Lambda instance.
• Cold starts are problematic due to slow database connections and limited database connection management.
• Serverless architecture leads to many servers needing individual connections to the database, complicating resource management.

Serverless databases optimizing connection management, reducing compute usage by 50%, and enabling multiple invocations with Cloudflare's idle time billing model

• Serverless databases allow for faster connection management, reducing the need for heavy pooling systems.
• Efficiency gains have been reported, with some private beta customers seeing up to a 50% reduction in compute usage.
• The traditional serverless model ties each invocation to a single function instance, causing idle time.
• The new model enables a single instance to handle multiple invocations, utilizing idle time to process additional requests.
• Cloudflare workers operate differently by not billing for idle time, using V8 isolates to handle multiple requests simultaneously within a single server.

Cloudflare's DV8 isolate model enabling simultaneous application deployment with idle time costs, restricted feature access, and enhanced Node.js concurrency utilization

• Cloudflare allows multiple developers to serve applications simultaneously on the same server using the DV8 isolate model, but each server is dedicated to a single user's code while it is running.
• Unlike AWS's low latency runtime, Cloudflare's model keeps the server up for a user’s code even when not running, leading to potential idle time costs.
• The V8 isolate model restricts access to certain features like file system operations, preventing interference between different users' code.
• Concurrency in traditional VPS happens entirely in Node, whereas in serverless models like AWS Lambda and Vercel, concurrency management occurs through separate server instances or additional concurrency layers.
• Node.js was designed for asynchronous IO concurrency, but serverless functions often fail to utilize available CPU fully during single invocations.
• The new concurrency model in Vercel functions allows for better utilization of Node's concurrency capabilities, enabling handling multiple requests while waiting for backend responses.
• The efficiency gains from the new concurrency model are not fully realized in applications with low request rates, as evidenced by the limited savings observed in the Ping application deployment.

Optimizing compute cost savings through usage patterns, concurrency management, and performance trade-offs

• Users can save significant amounts on compute costs, sometimes as high as 50%, by optimizing their usage on the platform.
• Savings are more pronounced for users consuming a high number of compute hours; the more hours consumed, the greater the savings.
• There are trade-offs in the current implementation during the beta phase, including limits on concurrent invocations which may affect latency for CPU-bound workloads.
• If a function's execution time is longer than the I/O wait time, it may block subsequent requests, impacting performance negatively.
• The benefits of concurrency and pooled connections depend on the balance between compute time and waiting time; if waiting time is minimal, the savings from the optimizations may be reduced.
• The platform is working on improving the detection of unhealthful concurrency to ensure optimized performance in appropriate situations.

Concurrent invocations in serverless architecture, memory leak tolerance, stateless code with external state management, efficiency enhancements without core nature changes, cost reductions through optimized idle compute usage, and internal isolation mechanisms in Vercel's serverless functions

• A single node process handling multiple invocations concurrently is a significant change in how Node is typically run in production.
• Memory leaks can be tolerated in serverless environments because the server stops running before they become problematic.
• Writing stateless code that stores state externally (e.g., in a database) is recommended for serverless applications to avoid issues with concurrent requests.
• The new architecture enhances the efficiency of serverless functions without altering their core serverless nature, such as automatic scaling and maintenance.
• Vercel's functions can achieve significant cost reductions through optimized compute usage during idle times without requiring code changes.
• Other platforms like Google Cloud Run, Doo Deo, and Cloudflare Workers also innovate on concurrent invocations for serverless compute, but Vercel's approach is distinguished by its internal isolation mechanisms.

Serverless architecture efficiency gains, server-side data management, optimized API requests, and dynamic application streaming

• The new feature in serverless architecture shows a significant efficiency gain and reduced compute costs for many users.
• React's new architecture encourages developers to keep data management on the server side and the client side lean.
• The shift to server-side management results in more API requests but can also lead to fewer overall requests when optimized properly.
• A powerful dynamic application can be achieved with fewer requests by streaming additional data after an initial authentication.
• The new model can reduce repeated work by minimizing the number of invocations needed during a page load.
• Server components can lead to better data handling and help avoid stale state issues common in single-page applications.
• Different traffic patterns arise from server component patterns compared to traditional client-side calling methods.
• The Next.js app router supports server-side React and nested routes, promoting an optimized full-stack architecture.

Granular routing requirements for serverless applications, caching strategies for dynamic scalability, and the impact of Next.js deployment on server demands

• More granular routes are needed to mirror data requirements in serverless applications, especially when combined with app routers, leading to increased server traffic and reconstruction of the server tree.
• Fair mitigates increased server needs with sophisticated caching and component rendering, but scalability for dynamic apps requires support from distributed deployment infrastructure.
• Deploying Next.js on Vercel involves using Vercel functions, which highlights the limitations of app routers due to increased server demands.
• Many developers use Next.js on Vercel as a slightly enhanced Create React App, often hitting APIs unrelated to Vercel for data.
• The significant cost savings from function currencies primarily benefit users with slow databases or external AI services, rather than large Next.js apps rendering numerous React components.
• The alignment between React, Next.js, and Vercel reflects the challenges posed by React's new architecture, which embraces the server-client model.
• The cost of rendering large apps and recreating HTML isn't solved by concurrency models; it relies on the speed of external data requests.
• The shift in waiting for API requests from the client to the server means that developers now pay for the compute time when waiting for responses.
• Node on Lambda blocks while waiting, and if concurrency is elevated to the server level, it can optimize performance.
• React developers are disproportionately impacted by these changes due to their reliance on new APIs and serverless architectures.

Empowering React Developers through Infer-Level Problem Solving for an Effortless Future

• many react devs haven't and by solving these problems at an infer level you can do the right thing more easily
• it's not simple but it is really exciting
• the future that it enables is hopefully one where we don't have to think about any of the stuff at all