The importance of participation in academic conferences is well known for members of the scientific community. It is not only for the feedback and the improvement of the work, it is also about career development, building networks and increasing visibility. Nevertheless, women continue to be under-represented in these academic events and even more so in the most visible positions such as speaking roles. This paper presents the development of a tool based on performance indicators, which will allow monitoring and evaluating gender roles and inequalities in academic conferences in order to tackle the underrepresentation of women. The study identifies relevant perspectives (participation, organizational structure and attitudes) and designs specific lists of performance indicators for each of them. The tool is based on a combination of two multicriteria techniques, Analytic Hierarchy Process and Analytic Hierarchy Process Sort, and a qualitative analysis based on in-depth interviews and information gathered from a focus group. The use of the AHP multi-criteria decision technique has allowed us to weight the indicators according to the opinion of several experts, and with them to be able to generate from these weightings composite indicators for each of the three dimensions. The most relevant indicators were for the participation dimension. Additionally, the tool developed has been applied to an academic conference which has been monitored in real time. The results are shown as a traffic light visualization approach, where red means bad performance, yellow average performance and green good performance, helping us to present the results for each indicator. Finally, proposals for improvement actions addressed to the red indicators are explained. The work carried out highlights the need to broaden the study of gender equality in academic conferences, not only regarding the participation but also the performance of different roles and functions.