I would have added to his sources proving Jews sought to rebuild the Temple whenever they had the chance this:
"In 361 C.E. a dedicated pagan, known today as Julian the Apostate, became emperor. He ruled for two years and actively sought to bring back Roman paganism at the expense of Christianity. For a combination of religious and political reasons Julian decided to allow Jews to return to Jerusalem and ordered them to rebuild the Temple. The church father Sozomenous claims that he even provided 'public money' to fund the construction and renew the sacrificial cult. By reconstructing the Temple Julian may have sought to discredit Jesus' prophecy about its destruction, as Sozomenous claims in his Ecclesiastical History*; or he may have also been motivated by the strategic consideration that 'his enemy's enemy was his friend', and sought to develop a political alliance with the Jews.
Regardless of the motives, Jews returned to the city and, according to the church father Philostorgious, even erected a synagogue near the site of the Temple. Another church father, Rufinus, claims that Jews came to Jerusalem from 'every place and every province'. Construction began in earnest. Sozomenous states that the Jews brought 'skillful artisans, collected materials, cleared the ground...even the women carried heaps of earth, and brought their necklaces and other female ornaments towards defraying the expense'.
At this time we may find the beginnings of Jewish reverence for the Western Wall. Graffiti carved on the Western Wall bearing a messianic verse from Isaiah may have been written at this time...Construction did not continue for very long. In May 363 C.E. an earthquake, and subsequent fire, caused local officials to halt work on the Temple. At the same time Julian's death in a war with the Persians, and replacement by a Christian Emperor named Jovian, brought the restoration project to an end. Excavations conducted by Prof. Benjamin Mazar at the southwest corner of the Temple Mount, uncovered buildings containing a thick layer of ash, indicating their destruction by fire. Associated coins show that the last period of occupation for these buildings was at the end of Julian's reign (362-363 C.E.)."
Julian Proposes to Rebuild Jerusalem, 362-363:
To The Community Of The Jews
In times past, by far the most burdensome thing in the yoke of your slavery has been the fact that you were subjected to unauthorized ordinances and had to contribute an untold amount of money to the accounts of the treasury. [Ever since Vespasian, about 72 CE, the Jews had been paying the Romans special Jewish taxes, like the Fiscus Judaicus.] Of this I used to see many instances with my own eyes, and I have learned of more, by finding the records which are preserved against you. Moreover, when a tax was about to be levied on you again I prevented it, and compelled the impiety of such obloquy to cease here; and I threw into the fire the records against you that were stored in my desks; so that it is no longer possible for anyone to aim at you such a reproach of impiety. My brother [cousin] Constantius of honored memory [in whose reign, 337-361, severe laws were enacted against the Jews] was not so much responsible for these wrongs of yours as were the men who used to frequent his table, barbarians in mind, godless in soul. These I seized with my own hands and put them to death by thrusting them into the pit, that not even any memory of their destruction might still linger amongst us.
And since I wish that you should prosper yet more, I have admonished my brother Iulus [Hillel II, d. 365], your most venerable patriarch, that the levy which is said to exist among you [the taxes paid by world Jewry for support of the Palestinian patriarchate] should be prohibited, and that no one is any longer to have the power to oppress the masses of your people by such exactions; so that everywhere, during my reign, you may have security of mind, and in the enjoyment of peace may offer more fervid prayers for my reign to the Most High God, the Creator, who has deigned to crown me with his own immaculate right hand. For it is natural that men who are distracted by any anxiety should be hampered in spirit, and should not have so much confidence in raising their hands to pray; but that those who are in all respects free from care should rejoice with their whole hearts and offer their suppliant prayers on behalf of my imperial office to Mighty God, even to Him who is able to direct my reign to the noblest ends, according to my purpose.
This you ought to do, in order that, when I have successfully concluded the war with Persia, I may rebuild by my own efforts the sacred city of Jerusalem [closed to the Jews since Hadrian, 135 CE], which for so many years you have longed to see inhabited, and may bring settlers there, and, together with you, may glorify the Most High God therein.
II. The Failure To Rebuild the Temple, 363
Though the emperor hated and oppressed the Christians, he manifested benevolence and humanity towards the Jews. He wrote to the Jewish patriarchs and leaders, as well as to the people, requesting them to pray for him, and for the prosperity of the empire. In taking this step he was not actuated, I am convinced, by any respect for their religion; for he was aware that it is, so to speak, the mother of the Christian religion, and he knew that both religions rest upon the authority of the [biblical] patriarchs and the prophets; but he thought to grieve the Christians by favoring the Jews, who are their most inveterate enemies. But perhaps he also calculated upon persuading the Jews to embrace paganism and sacrifices; for they were only acquainted with the mere letter of Scripture, and could not, like the Christians and a few of the wisest among the Hebrews, discern the hidden meaning [the allegorical meaning, through which the Christians could prove the validity of Christianity from the Old Testament].
Events proved that this was his real motive; for he sent for some of the chiefs of the race and exhorted them to return to the observance of the laws of Moses and the customs of their fathers. On their replying that because the Temple in Jerusalem was overturned, it was neither lawful nor ancestral to do this in another place than the metropolis out of which they had been cast, he gave them public money, commanded them to rebuild the Temple, and to practice the cult similar to that of their ancestors, by sacrificing after the ancient way. [Sacrifice was permitted by Jewish law only in Jerusalem.] The Jews entered upon the undertaking, without reflecting that, according to the prediction of the holy prophets, it could not be accomplished. They sought for the most skillful artisans, collected materials, cleared the ground, and entered so earnestly upon the task, that even the women carried heaps of earth, and brought their necklaces and other female ornaments towards defraying the expense.
The emperor, the other pagans, and all the Jews, regarded every other undertaking as secondary in importance to this. Although the pagans were not well-disposed towards the Jews, yet they assisted them in this enterprise, because they reckoned upon its ultimate success, and hoped by this means to falsify the prophecies of Christ. [Since Jesus in the New Testament had prophesied the destruction of the Temple, its rebuilding would make of him a false prophet.] Besides this motive, the Jews themselves [relying on the sympathy of Julian] were impelled by the consideration that the time had arrived for rebuilding their Temple.
When they had removed the ruins of the former building, they dug up the ground and cleared away its foundation; it is said that on the following day when they were about to lay the first foundation, a great earthquake occurred, and by the violent agitation of the earth, stones were thrown up from the depths, by which those of the Jews who were engaged in the work were wounded, as likewise those who were merely looking on. The houses and public porticos, near the site of the Temple, in which they had diverted themselves, were suddenly thrown down; many were caught thereby, some perished immediately, others were found half dead and mutilated of hands or legs, others were injured in other parts of the body.
When God caused the earthquake to cease, the workmen who survived again returned to their task, partly because such was the edict of the emperor, and partly because they were themselves interested in the undertaking. Men often, in endeavoring to gratify their own passions, seek what is injurious to them, reject what would be truly advantageous, and are deluded by the idea that nothing is really useful except what is agreeable to them. When once led astray by this error, they are no longer able to act in a manner conducive to their own interests, or to take warning by the calamities which are visited upon them. [The Church Father here records his belief that the Temple could not be rebuilt.]
The Jews, I believe, were just in this state; for, instead of regarding this unexpected earthquake as a manifest indication that God was opposed to the reerection of their Temple, they proceeded to recommence the work. But all parties relate that they had scarcely returned to the undertaking, when fire burst suddenly from the foundations of the Temple, and consumed several of the workmen.