The following work analyzes the effect of the composition of a hemp-lime composite on key mechanical and physical properties. The article contains results from testing the compressive strength, vapor permeability, and thermal conductivity of the composite, depending on the composition of the mix. The mixes differed from each other in binder composition and in the proportion of binder to hemp shives. The obtained results were compared with the results from other scientific literature. Based on this, conclusions were drawn that the binder composition is of secondary importance for the analyzed physical and mechanical properties of the hemp-lime composite. The main property that determines the values of the thermal conductivity coefficient as well as the compression strength is the density of the material, which depends on the proportion of binder to aggregate and the level of compaction of the mix. The value of the diffusion resistance coefficient of the analyzed material was very low regardless of the composition of the composite.