Labor productivity in building construction has long been a focused research topic due to the high contribution of labor cost in the building total costs. This study, among a few studies that used scaled data that were collected directly from measuring equipment and onsite activities, utilized neural networks to model the productivity of two main construction tasks and influencing factors. The neural networks show their ability to predict the behaviors of labor productivity of the formwork and rebar tasks in a test case of a high-rise building. A multilayer perceptron that had two layers and used sigmoid as its activation function provided the best effectiveness in predicting the relations among data. Among eleven independent factors, weather (e.g., temperature, precipitation, sun) generally played the most important role while crew factors were distributed in the mid of the ranking and the site factor (working floor height) played a mild role. This study confirms the robustness of neural networks in productivity research problems and the importance of working environments to labor productivity in building construction. Managerial implications, including careful environmental factors and crew structure deliberation, evolved from the study when labor productivity improvement is considered.