- Published on
Architecture ntipatterns
- Authors
- Name
- Fortan Pireva
- @fortanpireva
The dominant architectural style today is the horizontally scaled farm of commodity hardware. Horizontal scaling - more servers that run the same application code.
Provides fault tolerance through redundancy.
Even though, in general horizontal clusters are not subject to the single point of failure, they can exhibit a load-related failure mode. (memory- leak from application code) Chain reaction occurs when an application that has some defect - load-related crach, resource leak. This can also be caused by blocked threads.
Things to remember
- One server down jeopardizes the rest
- Hunt for resource leaks.
- Hunt for obscure timing bugs - (race conditions)
- Use autoscaling - In cloud, creation of health checks for every auto-scaling group is a must
- Defend with Bulkheads
Cascading failures
Occurs when a crack in one layer triggers a crack in a calling layer.
This often result from resource pools that get dreained because of a failure in a lower layer. Integration points without timeouts are a surefire to create cascading failures.
Cascading failures are the number-one crack accelerator. The most effective patterns to combat cascading failures are Circuit Breaker and Timeouts.
To remember
- Stop cracks from jumping the gap
- Scrutinize resource pools
- Defend with Timeouts and Circuit Breaker.
Users
Human users have a gift for doing exactly the worst possible thing at the worst possible time.
Traffic
“Capacity” is the maximum throughput your system can sustain under a given workload while maintaining acceptable performance.
Heap memory - is a hard limit, particularly in managed code languages. The “On-heap memory user session”
Keep as little in the in-memory session as possible. Weak references - the weak reference holds another object, called the payload, but only until the garbage collector needs to reclaim memory. Usually the only guarantee is that weakly reachable objects will be reclaimed before an out-of-memory error occurs.
Another way to deal it with user memory - is to farm it out to a different process.
These approaches
exercise a trade-off between total addressable size and latency.
Sockets - Port number is 16-bit, 65536 total connections.
If there are only 64,511 ports available, and millions of connections how ?
- Virtual IP adresses OS binds additional IP addresses to the same network interface.
A bogon, is a wandering packet that got routed inefficiently and arrives late, possibly out of sequence, and after the connection is closed.
Expensive users - test aggresively. If retailer store expects 2% conversion rate, test for 4, 6 or 10% conversion rate.
Unwanted users
sessions are the Achilles’ heel of web applications. Pick a deep link from the site and start requesting it without sending cookies. Web servers never tell application servers that user stopped waiting for an answer.
Keeping out legitimate robots is fairly easy through use of robots.txt file
2 approches work
- technical - when identifying a scraper block it from the network
- legal
Denial of service (DDoS) attacks. Attacker causes computers widely distributed in the net, to. start generating load on your site. Load comes from a botnet.
A specialized circuit-breaker can help to limit the damage done by any particular host.
- Users consume memory
- Users do weird, random things
- Malicious users are out there
- Users will gang up on you
Blocked threads
There’s a catch about interpreted languages. The interpreter can be running, and the application can still be totally deadlocked, doing nothing useful.
The most common failure mode is navel gazing - a happily running interpreter with every single thread sitting around waiting for Godot.
In object theory, the Liskov subsitution principle states that any property that is true about objects of type T should also be true for objects of any subtype of T.
A method without side effects in the base class, should also be without side effects in the derived class.
Things to remember
- Recall that the Blocked Threads antipattern is the proximate cause of most failures.
- Scrutinize resource pools
- Use proven primitives
- Defend with Timeouts
- Beware of the code you can not see
Self-Denial Attacks
This type of attack is described as any situation in which the system - or the extended system that includes humans - conspires against itself.
Avoid this type of attack by building a “shared-nothing” architecture. ( Each server can run on it’s own without knowing what other server is doing)
Autoscaling can help when the traffic surge does arrive.
Things to remember
- Keep the lins of communication open
- Protect shared resources
- Expect rapid redistribution of any cool or valuable offer.
Scaling effects
Be sure to distinguish between point-to-point inside a service versus point-to-point between services. If the application will only ever have 2 servers, then point-to-point is fine.
Replacement potentials
- UDP broadcasts
- TCP or UDP broadcasts
- Publish/subscribe messaging
- Message queues
Shared resource
is some facility of all member of a horizonally scalable layer need to use. It could be a cluster manager, or a lock manager. When it gets overloaded it becomes a bottleneck.
The trouble with shared-nothing architecture is that it might scale better at the cost of failover.
Things to remember
- Examine production versus QA environments to spot Scaling Effects
- Watch out for point-to-point communication
- Watch out for shared resources
Callers and providers should be resilient, for the caller Circuit Breaker will help by relieving the pressure of downstream services when responses get slow or connections get refused. For the providers, Handshaking and Backpressure should be used to inform callers to throttle back on the requests.
Drive out Through Testing
Unbalanced capacities are rarely observed by QA, (scaled down to just 2 servers)
Things to remember:
- Examine server and thread counts
- Observe near Scaling Effects and users
- Vritualize QA and scale it up
- Stress both sides of the interface
Dogpile
When a bunch of servers impose this transient load all at once, it’s called a dogpile
occues in different situation:
- Booting up several servers, code upgrade or restart
- When cron job triggers at midnight
- When configuration managment system pushes out a change
Force multiplier
Like a lever, automation allows administrators to make large movements with less effort.
A service discovery service is a distributed system that attempts to report on the state of many distributed systems to other distributed systems.
“Control plane” refers to software that exists to help manage the infrastructure and applications rather than directly delivering user functionality.
A failure can also resiult when the “desired” state is computed incorrectly and may be impossible or impractical.
Things to apply in control plane software
- Apply hysterisis. Start machines quickly, but shout them down slowly.
Slow Responses
Generating a slow response is worse than refusing a connection or returning an error, particularly in the context of middle-layer services.
Slow responses usually come from exessive demand.
Things to remember
- Slow responses trigger Cascading Failures
- For websites, slow responses cause more traffic (reload button)
- Consider Fail Fast
- Hunt for memory leaks or resource contention
Unbounded result set
In the abstract, an unbounded result set occurs when the caller allows the system to dictate terms. It’s a failure in handshaking. Social media assumed at first that the number of connections per user would be distributed like a bell-curve, but it’s actually distributed like a power law.
Things to remember
- Use realistic data volumes
- Paginate at the front end
- Don’t rely on data procedures
- Put limits to other application-level protocols