<br> tags omitted

5accaf10 · pabvald · 7cd99d00 · 5accaf10 · 5accaf10 · 5accaf10
Commit 5accaf10 authored Mar 4, 2020 by pabvald
--- a/generate_qa_pairs.py
+++ b/generate_qa_pairs.py
@@ -111,8 +111,8 @@ def extract_qa_pairs_intrel(page):
        q_spans = reduce(lambda a, b: a+b, list(map(lambda x : x.find_parents('span'), q_is)))
        sec_questions = list(map(lambda x : x.get_text(), q_spans))

-        sec_answers = list(map(lambda x : str(x.contents[0]).replace('<p>', '').replace('</p>', '')
-                        .replace('</p>', ' ').replace('<ul>', '').replace('</ul>', '').replace('<li>', ' ').replace('</li>', ','), 
+        sec_answers = list(map(lambda x : str(x.contents[0]).replace('<p>', '').replace('</p>', '').replace('</p>', ' ')
+                        .replace('<ul>', '').replace('</ul>', '').replace('<li>', ' ').replace('</li>', ',').replace('<br/>', ' '), 
                        sec_contents[sec].find_all('div', class_=['eael-accordion-content', 'clearfix'])))

        for p in range(len(sec_questions)):
@@ -143,9 +143,9 @@ def extract_qa_pairs_doctorate(page):

    #process_a_tags(sec_content)

-    sec_answers = list(map(lambda x: str(x.contents[1].contents[0]).replace('<p style=\"text-align: justify;\">', '')
-                    .replace('</p>', ' ').replace('<ul>', '').replace('</ul>', '').replace('<li>', ' ').replace('</li>', ',')
-                    ,sec_content.find_all('div', class_='panel-body')))
+    sec_answers = list(map(lambda x: str(x.contents[1].contents[0]).replace('<p style=\"text-align: justify;\">', '').replace('</p>', ' ')
+                    .replace('<ul>', '').replace('</ul>', '').replace('<li>', ' ').replace('</li>', ',').replace('<br/>', ' '),
+                    sec_content.find_all('div', class_='panel-body')))
    
    for p in range(len(sec_questions)):
        qa_pairs.append({'question': sec_questions[p], 

--- a/intrel_qa_pairs.csv
+++ b/intrel_qa_pairs.csv
@@ -22,14 +22,14 @@ I SPEAK SPANISH BUT DO NOT HAVE CERTIFICATES. WHAT CAN I DO?,"You can send us a
 WHAT DO I HAVE TO DO THE FIRST DAY I GET TO THE UVA?," If you are going to study in the city of Valladolid, you should go to the International Relations Office to register that you have arrived and we will give you information. You can find our address and the office opening hours through this link., If you are going to study in Palencia, Segovia, or Soria you should visit your coordinating teacher, who will register your arrival and will give you academic information. You should let your coordinating teacher know in advance by email the date on which you will be arriving. You can find their contact details in your Personal Area.,If your coordinating teacher is late in answering you, you can also contact the assistant at the International Relations Office on your campus through the following email addresses: Palencia Campus: <a href=""mailto:internacional.palencia@uva.es"">palencia@uva.es</a>, Segovia Campus: <a href=""mailto:internacional.segovia@uva.es"">segovia@uva.es</a>, Soria Campus: <a href=""mailto:internacional.soria@uva.es"">soria@uva.es</a>,"
 I STILL DO NOT HAVE MY STUDENT CARD AND I NEED IT. WHAT SHOULD I DO?,"You can still have access to all university services and facilities even if you are still waiting for your student card.There are two ways to access: with your enrolment sheet or downloading the UVa app for smartphones, available for <a href=""https://play.google.com/store/apps/details?id=com.pocketuniversity.uva"" rel=""noopener"" target=""_blank"">Android</a> and <a href=""https://itunes.apple.com/app/id1031505989"" rel=""noopener"" target=""_blank"">iOS</a>."
 WHAT MARK/GRADE WILL I GET IN MY COUNTRY WITH THE EXAMINATIONS I TAKE AT THE UNIVERSITY OF VALLADOLID?,"Each country has a different grading system. European countries share a transfer system which you can check out in Wikipedia: <a href=""https://es.wikipedia.org/wiki/European_Credit_Transfer_and_Accumulation_System"" rel=""noopener"" target=""_blank"">ECTS</a>, <a href=""https://en.wikipedia.org/wiki/ECTS_grading_scale"" rel=""noopener"" target=""_blank"">Grading scale (only in English)</a>."
-WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?,"First term examinations are held in January, although some teachers may bring theirs forward to late December.<br/>Second term examinations are held in June."
+WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?,"First term examinations are held in January, although some teachers may bring theirs forward to late December.  Second term examinations are held in June."
 I AM NOT IN SPAIN ON THE DAY OF THE EXAM. CAN I TAKE THE EXAM IN MY HOME COUNTRY?,You should ask your course teacher. They will tell you if there is another way for you to take the exam.
 I DID NOT PASS AN EXAM. WHAT CAN I DO?,"In Valladolid, you have the chance to take an exam again if you do not pass. These exams are known as “<em>extraordinarios</em>”."
 WHEN ARE THE EXTRAORDINARIO EXAMS?,<em>Extraordinario</em> exams for the first term are held in late January or early February. The <em>extraordinario</em> exams for the second term are held in late June.
 I PASSED AN EXAM BUT WITH A LOW MARK/GRADE AND I WANT TO IMPROVE IT. CAN I DO SO?,"You should ask your course teacher; some allow you to do an assignment in order to increase your mark/grade; others allow you to take the exam again, others do not allow you to improve your mark. Your teacher must tell you what will happen if they allow you to take the exam again and you fail the second time.You should never take the <em>extraordinario</em> exam without first speaking to your teacher."
 I AM GOING TO TAKE AN EXAM AGAIN. WHAT MUST I DO?," If they are first term exams and you are leaving in the second term, nothing., If they are first term exams and you are leaving in February or if they are second term exams, you should do the following: As soon as you know that you need to take at least one <em>extraordinario</em> exam, send an email to <a href=""mailto:incoming@uva.es"">incoming@uva.es</a> or <a href=""mailto:raquel.chico@uva.es"">chico@uva.es</a> to let us know that you will be doing <em>extraordinario</em> exams., During your final <em>extraordinario</em> exam, you should ask the teacher invigilating for a proof of attendance note (<em>justificante de asistencia</em>)., After your final exam, you should take the proof of attendance note to the International Relations Office, or to your coordinating teacher if you are studying in Palencia, Segovia or Soria.,,"
 "HOW WILL I RECEIVE MY MARKS/GRADES CERTIFICATE, TRANSCRIPT OF RECORDS, DOCUMENT AFTER THE MOBILITY?",The International Relations Office will send it to you by email.
-MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?,"No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.<br/>If this is a problem for your university, you should contact us directly."
+MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?,"No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.  If this is a problem for your university, you should contact us directly."
 WHAT MUST I DO WHEN I FINISH MY STUDIES AT THE UNIVERSITY OF VALLADOLID?,"You should pick up the certificate accrediting your stay (<em>certificado de estancia</em>) or Letter 2 at the same place where you received your certificate of arrival or Letter 1: If you have studied in the city of Valladolid, you should go to the International Relations Office. , If you have studied in Palencia, Segovia, or Soria, you should see your coordinating teacher.,The International Relations Office will send you your marks/grades certificate or transcript of records directly. "
 HOW CAN I EXTEND MY STAY AT THE UVA?,"You should fill in the following document (<a href=""http://relint.uva.es/wp-content/uploads/2019/02/DOCUMENTO_AMPLIACION_MATRICULA.pdf"" rel=""noopener"" target=""_blank"">download here)</a> and send it by email to <a href=""mailto:incoming@uva.es"">incoming@uva.es</a> or hand it in personally at the International Relations Office."
 MY UNIVERSITY WILL NOT GIVE ME AN EXTENSION. IS THERE ANOTHER WAY I CAN STAY AT THE UVA?,"Yes. Through the Visiting Students Programme, any university student may stay at the University of Valladolid for one or two terms. "
--- a/intrel_qa_pairs.json
+++ b/intrel_qa_pairs.json
@@ -92,7 +92,7 @@
        "question": "WHAT MARK/GRADE WILL I GET IN MY COUNTRY WITH THE EXAMINATIONS I TAKE AT THE UNIVERSITY OF VALLADOLID?"
    },
    {
-        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December.<br/>Second term examinations are held in June.",
+        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December.  Second term examinations are held in June.",
        "question": "WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?"
    },
    {
@@ -120,7 +120,7 @@
        "question": "HOW WILL I RECEIVE MY MARKS/GRADES CERTIFICATE, TRANSCRIPT OF RECORDS, DOCUMENT AFTER THE MOBILITY?"
    },
    {
-        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.<br/>If this is a problem for your university, you should contact us directly.",
+        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.  If this is a problem for your university, you should contact us directly.",
        "question": "MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?"
    },
    {

--- a/qa_pairs.csv
+++ b/qa_pairs.csv
@@ -22,14 +22,14 @@ I SPEAK SPANISH BUT DO NOT HAVE CERTIFICATES. WHAT CAN I DO?,"You can send us a
 WHAT DO I HAVE TO DO THE FIRST DAY I GET TO THE UVA?," If you are going to study in the city of Valladolid, you should go to the International Relations Office to register that you have arrived and we will give you information. You can find our address and the office opening hours through this link., If you are going to study in Palencia, Segovia, or Soria you should visit your coordinating teacher, who will register your arrival and will give you academic information. You should let your coordinating teacher know in advance by email the date on which you will be arriving. You can find their contact details in your Personal Area.,If your coordinating teacher is late in answering you, you can also contact the assistant at the International Relations Office on your campus through the following email addresses: Palencia Campus: <a href=""mailto:internacional.palencia@uva.es"">palencia@uva.es</a>, Segovia Campus: <a href=""mailto:internacional.segovia@uva.es"">segovia@uva.es</a>, Soria Campus: <a href=""mailto:internacional.soria@uva.es"">soria@uva.es</a>,"
 I STILL DO NOT HAVE MY STUDENT CARD AND I NEED IT. WHAT SHOULD I DO?,"You can still have access to all university services and facilities even if you are still waiting for your student card.There are two ways to access: with your enrolment sheet or downloading the UVa app for smartphones, available for <a href=""https://play.google.com/store/apps/details?id=com.pocketuniversity.uva"" rel=""noopener"" target=""_blank"">Android</a> and <a href=""https://itunes.apple.com/app/id1031505989"" rel=""noopener"" target=""_blank"">iOS</a>."
 WHAT MARK/GRADE WILL I GET IN MY COUNTRY WITH THE EXAMINATIONS I TAKE AT THE UNIVERSITY OF VALLADOLID?,"Each country has a different grading system. European countries share a transfer system which you can check out in Wikipedia: <a href=""https://es.wikipedia.org/wiki/European_Credit_Transfer_and_Accumulation_System"" rel=""noopener"" target=""_blank"">ECTS</a>, <a href=""https://en.wikipedia.org/wiki/ECTS_grading_scale"" rel=""noopener"" target=""_blank"">Grading scale (only in English)</a>."
-WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?,"First term examinations are held in January, although some teachers may bring theirs forward to late December.<br/>Second term examinations are held in June."
+WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?,"First term examinations are held in January, although some teachers may bring theirs forward to late December.  Second term examinations are held in June."
 I AM NOT IN SPAIN ON THE DAY OF THE EXAM. CAN I TAKE THE EXAM IN MY HOME COUNTRY?,You should ask your course teacher. They will tell you if there is another way for you to take the exam.
 I DID NOT PASS AN EXAM. WHAT CAN I DO?,"In Valladolid, you have the chance to take an exam again if you do not pass. These exams are known as “<em>extraordinarios</em>”."
 WHEN ARE THE EXTRAORDINARIO EXAMS?,<em>Extraordinario</em> exams for the first term are held in late January or early February. The <em>extraordinario</em> exams for the second term are held in late June.
 I PASSED AN EXAM BUT WITH A LOW MARK/GRADE AND I WANT TO IMPROVE IT. CAN I DO SO?,"You should ask your course teacher; some allow you to do an assignment in order to increase your mark/grade; others allow you to take the exam again, others do not allow you to improve your mark. Your teacher must tell you what will happen if they allow you to take the exam again and you fail the second time.You should never take the <em>extraordinario</em> exam without first speaking to your teacher."
 I AM GOING TO TAKE AN EXAM AGAIN. WHAT MUST I DO?," If they are first term exams and you are leaving in the second term, nothing., If they are first term exams and you are leaving in February or if they are second term exams, you should do the following: As soon as you know that you need to take at least one <em>extraordinario</em> exam, send an email to <a href=""mailto:incoming@uva.es"">incoming@uva.es</a> or <a href=""mailto:raquel.chico@uva.es"">chico@uva.es</a> to let us know that you will be doing <em>extraordinario</em> exams., During your final <em>extraordinario</em> exam, you should ask the teacher invigilating for a proof of attendance note (<em>justificante de asistencia</em>)., After your final exam, you should take the proof of attendance note to the International Relations Office, or to your coordinating teacher if you are studying in Palencia, Segovia or Soria.,,"
 "HOW WILL I RECEIVE MY MARKS/GRADES CERTIFICATE, TRANSCRIPT OF RECORDS, DOCUMENT AFTER THE MOBILITY?",The International Relations Office will send it to you by email.
-MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?,"No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.<br/>If this is a problem for your university, you should contact us directly."
+MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?,"No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.  If this is a problem for your university, you should contact us directly."
 WHAT MUST I DO WHEN I FINISH MY STUDIES AT THE UNIVERSITY OF VALLADOLID?,"You should pick up the certificate accrediting your stay (<em>certificado de estancia</em>) or Letter 2 at the same place where you received your certificate of arrival or Letter 1: If you have studied in the city of Valladolid, you should go to the International Relations Office. , If you have studied in Palencia, Segovia, or Soria, you should see your coordinating teacher.,The International Relations Office will send you your marks/grades certificate or transcript of records directly. "
 HOW CAN I EXTEND MY STAY AT THE UVA?,"You should fill in the following document (<a href=""http://relint.uva.es/wp-content/uploads/2019/02/DOCUMENTO_AMPLIACION_MATRICULA.pdf"" rel=""noopener"" target=""_blank"">download here)</a> and send it by email to <a href=""mailto:incoming@uva.es"">incoming@uva.es</a> or hand it in personally at the International Relations Office."
 MY UNIVERSITY WILL NOT GIVE ME AN EXTENSION. IS THERE ANOTHER WAY I CAN STAY AT THE UVA?,"Yes. Through the Visiting Students Programme, any university student may stay at the University of Valladolid for one or two terms. "

--- a/qa_pairs.json
+++ b/qa_pairs.json
@@ -92,7 +92,7 @@
        "question": "WHAT MARK/GRADE WILL I GET IN MY COUNTRY WITH THE EXAMINATIONS I TAKE AT THE UNIVERSITY OF VALLADOLID?"
    },
    {
-        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December.<br/>Second term examinations are held in June.",
+        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December. Second term examinations are held in June.",
        "question": "WHEN ARE EXAMINATIONS HELD AT THE UNIVERSITY OF VALLADOLID?"
    },
    {
@@ -120,7 +120,7 @@
        "question": "HOW WILL I RECEIVE MY MARKS/GRADES CERTIFICATE, TRANSCRIPT OF RECORDS, DOCUMENT AFTER THE MOBILITY?"
    },
    {
-        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.<br/>If this is a problem for your university, you should contact us directly.",
+        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid. If this is a problem for your university, you should contact us directly.",
        "question": "MY UNIVERSITY HAS A DOCUMENT FOR THIS MARKS/GRADES CERTIFICATE. SHOULD I TAKE IT TO MY COORDINATING TEACHER TO HAVE IT SIGNED?"
    },
    {

--- a/qa_pairs_em.json
+++ b/qa_pairs_em.json
@@ -7038,7 +7038,7 @@
        "question": "WHAT MARK/GRADE WILL I GET IN MY COUNTRY WITH THE EXAMINATIONS I TAKE AT THE UNIVERSITY OF VALLADOLID?"
    },
    {
-        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December.<br/>Second term examinations are held in June.",
+        "answer": "First term examinations are held in January, although some teachers may bring theirs forward to late December. Second term examinations are held in June.",
        "embedding": [
            -0.03145499527454376,
            -0.3897700011730194,
@@ -9180,7 +9180,7 @@
        "question": "HOW WILL I RECEIVE MY MARKS/GRADES CERTIFICATE, TRANSCRIPT OF RECORDS, DOCUMENT AFTER THE MOBILITY?"
    },
    {
-        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid.<br/>If this is a problem for your university, you should contact us directly.",
+        "answer": "No. The rules in our university require us to use our own certificate. This is created following a strict and safe protocol. It also has to be signed by the university authorities, the Vice-rector for International Relations, and the General Secretary. A transcript signed by your coordinating teacher at the University of Valladolid will be not be valid. If this is a problem for your university, you should contact us directly.",
        "embedding": [
            0.015148751437664032,
            -0.23063500225543976,