Overview
The NVIDIA A100 and H100 represent two generations of NVIDIA's data center GPU architecture, powering the majority of AI/ML training and inference workloads worldwide. As newer generations enter the market, both GPUs have become available on the secondary market at significantly reduced prices compared to new units.
-
NVIDIA A100 (Ampere Architecture): Released in 2020, the A100 was a revolutionary leap in AI computing with its multi-instance GPU (MIG) capability and massive memory bandwidth. It remains the workhorse of many AI training clusters and cloud providers.
-
NVIDIA H100 (Hopper Architecture): Released in 2022, the H100 delivers up to 4x performance improvement over the A100 for large language model training, thanks to its Transformer Engine and fourth-generation Tensor Cores.
Both GPUs are available in PCIe and SXM form factors, with the SXM variants offering higher power envelopes and NVLink connectivity for multi-GPU scaling.
Specifications Comparison
| Parameter | NVIDIA A100 | NVIDIA H100 | |--------------------------|---------------------------|----------------------------| | Architecture | Ampere | Hopper | | Process Node | 7nm (TSMC) | 4nm (TSMC) | | Memory | 40GB or 80GB HBM2e | 80GB HBM3 | | Memory Bandwidth | 1.6 TB/s (80GB) | 3.35 TB/s | | TDP | 250W (PCIe) / 400W (SXM) | 350W (PCIe) / 700W (SXM) | | Form Factor | PCIe / SXM4 | PCIe / SXM5 | | NVLink | 3rd Gen, 600 GB/s | 4th Gen, 900 GB/s | | Tensor Cores | 3rd Gen | 4th Gen with Transformer Engine | | Release Year | 2020 | 2022 |
Used Market Pricing
-
NVIDIA A100: Currently trading between $6,000–$10,000 for PCIe variants, with 80GB models commanding premiums over 40GB versions. SXM variants with NVLink are typically 20-30% higher.
-
NVIDIA H100: Ranges from $18,000–$28,000 depending on form factor, memory configuration, and warranty status. SXM5 variants with full NVLink support are at the higher end of this range.
Price Influencers:
- Form Factor: SXM modules command higher prices than PCIe cards due to their superior multi-GPU scaling capabilities.
- Memory Size: 80GB A100s are approximately 30% more expensive than 40GB models.
- Warranty: Units with remaining manufacturer warranty or third-party support contracts carry premiums of 15-25%.
- Supply Constraints: H100 availability remains tight due to continued strong demand from AI companies.
When to Choose Each
NVIDIA A100
- Budget-Conscious AI Training: Excellent price-to-performance for most training workloads.
- Inference at Scale: MIG capability allows splitting a single GPU among multiple inference workloads.
- Established Workflows: Proven compatibility with existing AI frameworks and optimized software stacks.
- Lower Power Environments: Lower TDP makes it suitable for data centers with power constraints.
NVIDIA H100
- Large Language Model Training: The Transformer Engine delivers exceptional performance for transformer-based models.
- Future-Proofing: Better support for emerging AI architectures and larger model sizes.
- High-Performance Computing: Superior FP8 and mixed-precision performance for HPC workloads.
- Multi-GPU Scaling: Enhanced NVLink bandwidth benefits large-scale distributed training.
Secondary Market Availability
-
NVIDIA A100: High availability due to cloud provider refresh cycles and enterprise upgrades. Many units entering the market from hyperscaler decommissioning.
-
NVIDIA H100: Limited availability as the GPU is still relatively new and in high demand. Most used units come from failed startup liquidations or early enterprise refresh cycles.
Sourcing Challenges:
- Authenticity Verification: Counterfeit or modified GPUs are a concern; purchase from reputable dealers.
- Cooling Requirements: SXM variants require specific server platforms (DGX, HGX, or compatible OEM servers).
- Software Licensing: Some AI enterprise software licenses are tied to specific GPU generations.
- Power Infrastructure: H100 SXM5 systems require substantial power and cooling upgrades compared to A100 deployments.
Verdict
For most AI training and inference workloads, the NVIDIA A100 offers exceptional value on the used market. Its mature software ecosystem, proven reliability, and significantly lower cost make it ideal for budget-conscious organizations building AI infrastructure.
The NVIDIA H100 is worth the premium for organizations training large foundation models (100B+ parameters) or requiring maximum performance per watt. Its Transformer Engine and superior memory bandwidth deliver measurable productivity gains for cutting-edge AI research.
Recommendation:
- NVIDIA A100: Best for startups, academic institutions, and enterprises building cost-effective AI infrastructure.
- NVIDIA H100: Recommended for AI research labs, large-scale training operations, and organizations where training time directly impacts business outcomes.
Consider that 2-3 A100s often deliver better aggregate performance per dollar than a single H100 for workloads that parallelize well across multiple GPUs.