Predicting human intention in vehicles, pedestrians and bicyclists interactions can help autonomous vehicles and human drivers to plan their routes in a safer manner and better optimise the use of road space. Several studies have studied human intention when interacting with other agents at crossroads using handcrafted features, motif analyses, and machine learning approaches. Yet, many of them are limited in accuracy due to relatively insufficient consideration of surrounding agents and limited observations (occlusions and inaccurate estimation of pose and location) confined by camera angles. This study utilised a multi-branch Gated Recurrent Unit encoder-decoder (MBGED) model to predict the intention of pedestrians and bicyclists when contenting with vehicles at intersections by analysing the properties of directly and indirectly involved road agents. This study identified decisive factors of human intention and constructed an encoder-decoder architecture based on those factors. The network was trained, validated, and tested on unsignalised and uncontrolled intersections. The system predicted the intention of vulnerable road users with 96% accuracy, 91% precision, and 93% recall at 2 seconds before the intersections happen, which could provide a reliable reference for autonomous vehicle navigation and advanced driver assistant systems.